Oct 30 23:03:35.211007 ip-10-0-130-177 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 23:03:35.211171 ip-10-0-130-177 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 23:03:35.211274 ip-10-0-130-177 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 23:03:35.211800 ip-10-0-130-177 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 23:03:45.460371 ip-10-0-130-177 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 23:03:45.460388 ip-10-0-130-177 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 551ed6a1ea6a4c06a2a83baec0ac93c0 -- Oct 30 23:05:37.658406 ip-10-0-130-177 systemd[1]: Starting Kubernetes Kubelet... Oct 30 23:05:38.024314 ip-10-0-130-177 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:05:38.024314 ip-10-0-130-177 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 23:05:38.024314 ip-10-0-130-177 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:05:38.024314 ip-10-0-130-177 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:05:38.028094 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.027980 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:05:38.028094 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028095 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:05:38.028159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028100 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:05:38.028159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028103 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:05:38.028159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028107 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:05:38.028159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028110 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:05:38.028159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028112 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:05:38.028159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028115 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:05:38.028159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028118 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:05:38.028159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028121 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:05:38.028159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028125 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:05:38.028159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028129 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:05:38.028159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028133 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:05:38.028159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028137 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:05:38.028159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028140 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:05:38.028159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028143 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:05:38.028159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028145 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:05:38.028159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028148 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:05:38.028159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028151 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:05:38.028159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028154 2570 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:05:38.028159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028156 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:05:38.028629 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028159 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:05:38.028629 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028161 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:05:38.028629 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028164 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:05:38.028629 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028167 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:05:38.028629 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028170 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:05:38.028629 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028173 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:05:38.028629 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028175 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:05:38.028629 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028178 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:05:38.028629 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028180 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:05:38.028629 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028183 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:05:38.028629 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028186 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:05:38.028629 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028188 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:05:38.028629 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028191 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:05:38.028629 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028193 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:05:38.028629 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028196 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:05:38.028629 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028198 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:05:38.028629 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028201 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:05:38.028629 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028203 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:05:38.028629 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028208 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:05:38.029092 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028212 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:05:38.029092 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028215 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:05:38.029092 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028218 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:05:38.029092 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028221 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:05:38.029092 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028223 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:05:38.029092 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028226 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:05:38.029092 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028229 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:05:38.029092 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028231 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:05:38.029092 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028234 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:05:38.029092 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028237 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:05:38.029092 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028239 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:05:38.029092 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028242 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:05:38.029092 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028244 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:05:38.029092 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028247 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:05:38.029092 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028251 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:05:38.029092 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028254 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:05:38.029092 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028257 2570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:05:38.029092 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028259 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:05:38.029092 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028262 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:05:38.029092 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028265 2570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:05:38.029591 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028267 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:05:38.029591 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028270 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:05:38.029591 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028272 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:05:38.029591 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028275 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:05:38.029591 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028278 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:05:38.029591 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028280 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:05:38.029591 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028283 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:05:38.029591 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028285 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:05:38.029591 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028288 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:05:38.029591 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028290 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:05:38.029591 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028293 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:05:38.029591 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028295 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:05:38.029591 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028299 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:05:38.029591 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028302 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:05:38.029591 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028305 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:05:38.029591 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028307 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:05:38.029591 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028310 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:05:38.029591 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028312 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:05:38.029591 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028315 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:05:38.030029 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028318 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:05:38.030029 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028320 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:05:38.030029 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028322 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:05:38.030029 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028325 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:05:38.030029 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028327 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:05:38.030029 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028330 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:05:38.030029 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028333 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:05:38.030029 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028336 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:05:38.030029 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028339 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:05:38.030029 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028343 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:05:38.030029 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028347 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:05:38.030029 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028352 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:05:38.030029 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028355 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:05:38.030029 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028357 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:05:38.030029 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028360 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:05:38.030029 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028363 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:05:38.030029 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028366 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:05:38.030029 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028369 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:05:38.030029 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028372 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:05:38.030029 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028375 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:05:38.030513 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028377 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:05:38.030513 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028380 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:05:38.030513 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028382 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:05:38.030513 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028385 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:05:38.030513 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028387 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:05:38.030513 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028390 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:05:38.030513 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.028393 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:05:38.030513 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029184 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:05:38.030513 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029192 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:05:38.030513 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029195 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:05:38.030513 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029198 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:05:38.030513 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029201 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:05:38.030513 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029203 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:05:38.030513 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029206 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:05:38.030513 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029208 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:05:38.030513 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029212 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:05:38.030513 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029214 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:05:38.030513 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029217 2570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:05:38.030513 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029220 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:05:38.030513 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029224 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:05:38.030976 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029227 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:05:38.030976 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029230 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:05:38.030976 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029232 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:05:38.030976 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029236 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:05:38.030976 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029238 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:05:38.030976 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029241 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:05:38.030976 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029243 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:05:38.030976 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029246 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:05:38.030976 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029248 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:05:38.030976 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029251 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:05:38.030976 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029253 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:05:38.030976 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029256 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:05:38.030976 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029258 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:05:38.030976 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029261 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:05:38.030976 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029263 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:05:38.030976 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029266 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:05:38.030976 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029269 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:05:38.030976 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029272 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:05:38.030976 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029274 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:05:38.030976 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029278 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:05:38.031466 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029280 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:05:38.031466 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029283 2570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:05:38.031466 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029285 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:05:38.031466 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029288 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:05:38.031466 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029290 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:05:38.031466 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029293 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:05:38.031466 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029295 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:05:38.031466 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029298 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:05:38.031466 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029301 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:05:38.031466 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029303 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:05:38.031466 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029306 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:05:38.031466 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029309 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:05:38.031466 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029312 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:05:38.031466 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029314 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:05:38.031466 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029317 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:05:38.031466 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029319 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:05:38.031466 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029322 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:05:38.031466 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029324 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:05:38.031466 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029327 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:05:38.031917 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029329 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:05:38.031917 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029332 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:05:38.031917 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029334 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:05:38.031917 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029337 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:05:38.031917 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029339 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:05:38.031917 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029342 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:05:38.031917 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029344 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:05:38.031917 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029347 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:05:38.031917 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029350 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:05:38.031917 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029352 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:05:38.031917 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029354 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:05:38.031917 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029357 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:05:38.031917 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029360 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:05:38.031917 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029363 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:05:38.031917 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029366 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:05:38.031917 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029368 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:05:38.031917 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029371 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:05:38.031917 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029373 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:05:38.031917 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029375 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:05:38.032407 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029380 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:05:38.032407 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029384 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:05:38.032407 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029387 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:05:38.032407 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029389 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:05:38.032407 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029393 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:05:38.032407 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029396 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:05:38.032407 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029399 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:05:38.032407 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029401 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:05:38.032407 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029404 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:05:38.032407 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029406 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:05:38.032407 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029409 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:05:38.032407 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029411 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:05:38.032407 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029414 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:05:38.032407 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029417 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:05:38.032407 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029419 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:05:38.032407 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029423 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:05:38.032407 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029427 2570 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:05:38.032407 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029431 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:05:38.032407 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029434 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:05:38.032848 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029436 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:05:38.032848 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029439 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:05:38.032848 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029441 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:05:38.032848 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029444 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:05:38.032848 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029447 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:05:38.032848 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029449 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:05:38.032848 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029452 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:05:38.032848 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029454 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:05:38.032848 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029457 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:05:38.032848 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029460 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:05:38.032848 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029462 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:05:38.032848 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029465 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:05:38.032848 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029467 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:05:38.032848 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029470 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:05:38.032848 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029474 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:05:38.032848 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029477 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:05:38.032848 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029930 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:05:38.032848 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029935 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:05:38.032848 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029938 2570 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:05:38.032848 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029941 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:05:38.033327 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029944 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:05:38.033327 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029947 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:05:38.033327 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029949 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:05:38.033327 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029952 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:05:38.033327 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029955 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:05:38.033327 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029957 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:05:38.033327 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029960 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:05:38.033327 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029963 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:05:38.033327 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029965 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:05:38.033327 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029968 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:05:38.033327 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029971 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:05:38.033327 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029973 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:05:38.033327 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029976 2570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:05:38.033327 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029978 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:05:38.033327 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029981 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:05:38.033327 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029984 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:05:38.033327 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029988 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:05:38.033327 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029991 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:05:38.033327 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029993 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:05:38.033327 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029996 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:05:38.033795 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.029999 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:05:38.033795 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030002 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:05:38.033795 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030005 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:05:38.033795 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030007 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:05:38.033795 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030010 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:05:38.033795 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030013 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:05:38.033795 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030016 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:05:38.033795 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030018 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:05:38.033795 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030021 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:05:38.033795 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030023 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:05:38.033795 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030026 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:05:38.033795 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030029 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:05:38.033795 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030047 2570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:05:38.033795 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030050 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:05:38.033795 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030053 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:05:38.033795 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030058 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:05:38.033795 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030062 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:05:38.033795 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030065 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:05:38.033795 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030068 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:05:38.034259 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030071 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:05:38.034259 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030074 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:05:38.034259 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030076 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:05:38.034259 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030079 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:05:38.034259 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030082 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:05:38.034259 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030084 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:05:38.034259 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030087 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:05:38.034259 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030090 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:05:38.034259 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030094 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:05:38.034259 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030098 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:05:38.034259 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030101 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:05:38.034259 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030104 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:05:38.034259 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030107 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:05:38.034259 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030109 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:05:38.034259 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030112 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:05:38.034259 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030115 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:05:38.034259 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030118 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:05:38.034259 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030121 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:05:38.034259 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030124 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:05:38.034698 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030126 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:05:38.034698 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030129 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:05:38.034698 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030131 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:05:38.034698 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030134 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:05:38.034698 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030136 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:05:38.034698 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030139 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:05:38.034698 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030142 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:05:38.034698 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030144 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:05:38.034698 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030147 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:05:38.034698 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030149 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:05:38.034698 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030152 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:05:38.034698 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030155 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:05:38.034698 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030157 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:05:38.034698 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030160 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:05:38.034698 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030163 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:05:38.034698 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030165 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:05:38.034698 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030168 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:05:38.034698 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030170 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:05:38.034698 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030173 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:05:38.035155 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030176 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:05:38.035155 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030179 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:05:38.035155 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030181 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:05:38.035155 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030184 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:05:38.035155 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030187 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:05:38.035155 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030190 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:05:38.035155 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030192 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:05:38.035155 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030195 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:05:38.035155 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030197 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:05:38.035155 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030201 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:05:38.035155 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030203 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:05:38.035155 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030206 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:05:38.035155 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030209 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:05:38.035155 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030211 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:05:38.035155 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030214 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:05:38.035155 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030217 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:05:38.035155 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030219 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:05:38.035155 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030222 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:05:38.035155 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030225 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:05:38.035155 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030227 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:05:38.035625 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030230 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:05:38.035625 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030233 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:05:38.035625 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030235 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:05:38.035625 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030238 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:05:38.035625 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.030242 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:05:38.035625 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031104 2570 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 23:05:38.035625 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031114 2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 23:05:38.035625 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031120 2570 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 23:05:38.035625 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031124 2570 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 23:05:38.035625 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031128 2570 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 23:05:38.035625 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031131 2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 23:05:38.035625 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031135 2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 23:05:38.035625 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031139 2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 23:05:38.035625 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031142 2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 23:05:38.035625 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031145 2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 23:05:38.035625 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031149 2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 23:05:38.035625 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031153 2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 23:05:38.035625 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031156 2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 23:05:38.035625 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031159 2570 flags.go:64] FLAG: --cgroup-root="" Oct 30 23:05:38.035625 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031162 2570 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 23:05:38.035625 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031164 2570 flags.go:64] FLAG: --client-ca-file="" Oct 30 23:05:38.035625 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031167 2570 flags.go:64] FLAG: --cloud-provider="external" Oct 30 23:05:38.035625 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031171 2570 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 23:05:38.036162 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031176 2570 flags.go:64] FLAG: --cluster-domain="" Oct 30 23:05:38.036162 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031179 2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 23:05:38.036162 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031182 2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 23:05:38.036162 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031185 2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 23:05:38.036162 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031188 2570 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 23:05:38.036162 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031192 2570 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 23:05:38.036162 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031195 2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 23:05:38.036162 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031198 2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 23:05:38.036162 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031201 2570 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 23:05:38.036162 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031204 2570 flags.go:64] FLAG: --contention-profiling="false" Oct 30 23:05:38.036162 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031207 2570 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 23:05:38.036162 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031210 2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 23:05:38.036162 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031214 2570 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 23:05:38.036162 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031217 2570 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 23:05:38.036162 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031221 2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 23:05:38.036162 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031224 2570 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 23:05:38.036162 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031227 2570 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 23:05:38.036162 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031230 2570 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 23:05:38.036162 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031233 2570 flags.go:64] FLAG: --enable-server="true" Oct 30 23:05:38.036162 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031236 2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 23:05:38.036162 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031240 2570 flags.go:64] FLAG: --event-burst="100" Oct 30 23:05:38.036162 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031243 2570 flags.go:64] FLAG: --event-qps="50" Oct 30 23:05:38.036162 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031246 2570 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 23:05:38.036162 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031250 2570 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 23:05:38.036734 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031253 2570 flags.go:64] FLAG: --eviction-hard="" Oct 30 23:05:38.036734 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031257 2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 23:05:38.036734 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031260 2570 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 23:05:38.036734 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031263 2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 23:05:38.036734 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031266 2570 flags.go:64] FLAG: --eviction-soft="" Oct 30 23:05:38.036734 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031269 2570 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 23:05:38.036734 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031272 2570 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 23:05:38.036734 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031274 2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 23:05:38.036734 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031278 2570 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 23:05:38.036734 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031281 2570 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 23:05:38.036734 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031284 2570 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 23:05:38.036734 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031287 2570 flags.go:64] FLAG: --feature-gates="" Oct 30 23:05:38.036734 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031291 2570 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 23:05:38.036734 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031293 2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 23:05:38.036734 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031296 2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 23:05:38.036734 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031299 2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 23:05:38.036734 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031303 2570 flags.go:64] FLAG: --healthz-port="10248" Oct 30 23:05:38.036734 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031306 2570 flags.go:64] FLAG: --help="false" Oct 30 23:05:38.036734 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031308 2570 flags.go:64] FLAG: --hostname-override="ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:05:38.036734 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031312 2570 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 23:05:38.036734 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031315 2570 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 23:05:38.036734 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031318 2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 23:05:38.036734 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031321 2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 23:05:38.037296 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031325 2570 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 23:05:38.037296 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031327 2570 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 23:05:38.037296 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031330 2570 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 23:05:38.037296 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031333 2570 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 23:05:38.037296 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031336 2570 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 23:05:38.037296 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031339 2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 23:05:38.037296 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031342 2570 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 23:05:38.037296 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031344 2570 flags.go:64] FLAG: --kube-reserved="" Oct 30 23:05:38.037296 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031347 2570 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 23:05:38.037296 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031353 2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 23:05:38.037296 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031356 2570 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 23:05:38.037296 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031359 2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 23:05:38.037296 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031361 2570 flags.go:64] FLAG: --lock-file="" Oct 30 23:05:38.037296 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031364 2570 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 23:05:38.037296 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031367 2570 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 23:05:38.037296 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031370 2570 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 23:05:38.037296 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031375 2570 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 23:05:38.037296 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031378 2570 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 23:05:38.037296 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031382 2570 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 23:05:38.037296 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031385 2570 flags.go:64] FLAG: --logging-format="text" Oct 30 23:05:38.037296 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031387 2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 23:05:38.037296 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031391 2570 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 23:05:38.037296 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031393 2570 flags.go:64] FLAG: --manifest-url="" Oct 30 23:05:38.037296 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031396 2570 flags.go:64] FLAG: --manifest-url-header="" Oct 30 23:05:38.037296 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031400 2570 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 23:05:38.037925 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031403 2570 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 23:05:38.037925 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031407 2570 flags.go:64] FLAG: --max-pods="110" Oct 30 23:05:38.037925 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031410 2570 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 23:05:38.037925 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031413 2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 23:05:38.037925 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031416 2570 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 23:05:38.037925 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031419 2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 23:05:38.037925 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031422 2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 23:05:38.037925 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031425 2570 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 23:05:38.037925 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031428 2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 23:05:38.037925 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031435 2570 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 23:05:38.037925 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031438 2570 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 23:05:38.037925 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031441 2570 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 23:05:38.037925 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031444 2570 flags.go:64] FLAG: --pod-cidr="" Oct 30 23:05:38.037925 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031447 2570 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 23:05:38.037925 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031450 2570 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 23:05:38.037925 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031453 2570 flags.go:64] FLAG: --pods-per-core="0" Oct 30 23:05:38.037925 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031457 2570 flags.go:64] FLAG: --port="10250" Oct 30 23:05:38.037925 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031460 2570 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 23:05:38.037925 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031463 2570 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0ef590e33c2ab0754" Oct 30 23:05:38.037925 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031466 2570 flags.go:64] FLAG: --qos-reserved="" Oct 30 23:05:38.037925 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031469 2570 flags.go:64] FLAG: --read-only-port="10255" Oct 30 23:05:38.037925 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031472 2570 flags.go:64] FLAG: --register-node="true" Oct 30 23:05:38.037925 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031475 2570 flags.go:64] FLAG: --register-with-taints="" Oct 30 23:05:38.037925 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031479 2570 flags.go:64] FLAG: --registry-burst="10" Oct 30 23:05:38.037925 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031482 2570 flags.go:64] FLAG: --registry-qps="5" Oct 30 23:05:38.037925 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031484 2570 flags.go:64] FLAG: --reserved-cpus="" Oct 30 23:05:38.038578 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031489 2570 flags.go:64] FLAG: --reserved-memory="" Oct 30 23:05:38.038578 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031493 2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 23:05:38.038578 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031496 2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 23:05:38.038578 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031498 2570 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 23:05:38.038578 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031501 2570 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 23:05:38.038578 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031504 2570 flags.go:64] FLAG: --runonce="false" Oct 30 23:05:38.038578 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031507 2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 23:05:38.038578 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031510 2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 23:05:38.038578 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031513 2570 flags.go:64] FLAG: --seccomp-default="false" Oct 30 23:05:38.038578 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031516 2570 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 23:05:38.038578 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031521 2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 23:05:38.038578 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031524 2570 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 23:05:38.038578 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031527 2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 23:05:38.038578 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031530 2570 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 23:05:38.038578 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031533 2570 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 23:05:38.038578 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031536 2570 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 23:05:38.038578 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031539 2570 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 23:05:38.038578 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031542 2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 23:05:38.038578 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031545 2570 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 23:05:38.038578 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031548 2570 flags.go:64] FLAG: --system-cgroups="" Oct 30 23:05:38.038578 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031550 2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 23:05:38.038578 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031563 2570 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 23:05:38.038578 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031568 2570 flags.go:64] FLAG: --tls-cert-file="" Oct 30 23:05:38.038578 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031571 2570 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 23:05:38.038578 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031575 2570 flags.go:64] FLAG: --tls-min-version="" Oct 30 23:05:38.039446 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031578 2570 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 23:05:38.039446 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031580 2570 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 23:05:38.039446 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031583 2570 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 23:05:38.039446 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031586 2570 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 23:05:38.039446 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031603 2570 flags.go:64] FLAG: --v="2" Oct 30 23:05:38.039446 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031608 2570 flags.go:64] FLAG: --version="false" Oct 30 23:05:38.039446 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031615 2570 flags.go:64] FLAG: --vmodule="" Oct 30 23:05:38.039446 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031619 2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 23:05:38.039446 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.031623 2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 23:05:38.039446 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031748 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:05:38.039446 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031751 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:05:38.039446 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031755 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:05:38.039446 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031759 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:05:38.039446 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031762 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:05:38.039446 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031764 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:05:38.039446 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031767 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:05:38.039446 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031770 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:05:38.039446 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031774 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:05:38.039446 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031777 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:05:38.039446 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031780 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:05:38.039446 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031782 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:05:38.040172 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031785 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:05:38.040172 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031787 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:05:38.040172 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031790 2570 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:05:38.040172 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031793 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:05:38.040172 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031795 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:05:38.040172 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031798 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:05:38.040172 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031800 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:05:38.040172 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031803 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:05:38.040172 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031808 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:05:38.040172 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031812 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:05:38.040172 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031814 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:05:38.040172 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031817 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:05:38.040172 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031820 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:05:38.040172 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031823 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:05:38.040172 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031825 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:05:38.040172 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031828 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:05:38.040172 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031831 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:05:38.040172 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031833 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:05:38.040172 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031836 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:05:38.040172 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031839 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:05:38.040645 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031842 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:05:38.040645 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031845 2570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:05:38.040645 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031847 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:05:38.040645 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031850 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:05:38.040645 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031852 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:05:38.040645 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031855 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:05:38.040645 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031857 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:05:38.040645 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031860 2570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:05:38.040645 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031864 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:05:38.040645 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031867 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:05:38.040645 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031869 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:05:38.040645 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031872 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:05:38.040645 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031874 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:05:38.040645 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031877 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:05:38.040645 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031880 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:05:38.040645 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031882 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:05:38.040645 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031886 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:05:38.040645 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031890 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:05:38.040645 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031893 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:05:38.040645 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031896 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:05:38.041120 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031900 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:05:38.041120 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031902 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:05:38.041120 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031905 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:05:38.041120 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031907 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:05:38.041120 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031910 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:05:38.041120 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031912 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:05:38.041120 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031915 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:05:38.041120 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031917 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:05:38.041120 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031920 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:05:38.041120 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031923 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:05:38.041120 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031926 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:05:38.041120 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031928 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:05:38.041120 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031931 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:05:38.041120 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031934 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:05:38.041120 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031937 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:05:38.041120 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031940 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:05:38.041120 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031942 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:05:38.041120 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031944 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:05:38.041120 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031947 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:05:38.041710 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031949 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:05:38.041710 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031953 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:05:38.041710 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031956 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:05:38.041710 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031958 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:05:38.041710 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031961 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:05:38.041710 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031963 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:05:38.041710 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031966 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:05:38.041710 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031968 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:05:38.041710 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031971 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:05:38.041710 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031974 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:05:38.041710 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031977 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:05:38.041710 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031979 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:05:38.041710 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031982 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:05:38.041710 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031985 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:05:38.041710 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031988 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:05:38.041710 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031990 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:05:38.041710 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031993 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:05:38.041710 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031995 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:05:38.041710 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.031997 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:05:38.041710 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032000 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:05:38.042200 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032002 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:05:38.042200 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032005 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:05:38.042200 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032007 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:05:38.042200 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032010 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:05:38.042200 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032012 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:05:38.042200 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032015 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:05:38.042200 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032017 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:05:38.042200 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032020 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:05:38.042200 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032023 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:05:38.042200 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032025 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:05:38.042200 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032028 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:05:38.042200 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032030 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:05:38.042200 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032046 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:05:38.042200 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032050 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:05:38.042200 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032052 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:05:38.042200 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.032079 2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:05:38.042584 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032111 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:05:38.042584 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032115 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:05:38.042584 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032118 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:05:38.042584 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032120 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:05:38.042584 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032123 2570 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:05:38.042584 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032125 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:05:38.042584 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032128 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:05:38.042584 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032131 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:05:38.042584 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032133 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:05:38.042584 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032136 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:05:38.042584 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032141 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:05:38.042584 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032145 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:05:38.042584 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032148 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:05:38.042584 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032152 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:05:38.042584 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032154 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:05:38.042584 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032157 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:05:38.042584 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032160 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:05:38.042584 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032162 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:05:38.042584 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032165 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:05:38.042584 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032168 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:05:38.043159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032170 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:05:38.043159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032173 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:05:38.043159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032176 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:05:38.043159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032178 2570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:05:38.043159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032181 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:05:38.043159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032184 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:05:38.043159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032186 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:05:38.043159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032189 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:05:38.043159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032191 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:05:38.043159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032195 2570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:05:38.043159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032198 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:05:38.043159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032200 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:05:38.043159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032203 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:05:38.043159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032205 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:05:38.043159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032208 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:05:38.043159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032211 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:05:38.043159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032214 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:05:38.043159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032217 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:05:38.043159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032220 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:05:38.043159 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032224 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:05:38.043650 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032227 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:05:38.043650 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032229 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:05:38.043650 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032233 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:05:38.043650 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032236 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:05:38.043650 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032239 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:05:38.043650 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032241 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:05:38.043650 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032244 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:05:38.043650 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032246 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:05:38.043650 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032249 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:05:38.043650 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032252 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:05:38.043650 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032254 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:05:38.043650 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032257 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:05:38.043650 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032259 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:05:38.043650 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032262 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:05:38.043650 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032264 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:05:38.043650 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032267 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:05:38.043650 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032269 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:05:38.043650 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032272 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:05:38.043650 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032275 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:05:38.044112 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032277 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:05:38.044112 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032279 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:05:38.044112 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032284 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:05:38.044112 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032286 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:05:38.044112 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032289 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:05:38.044112 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032291 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:05:38.044112 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032294 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:05:38.044112 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032296 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:05:38.044112 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032299 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:05:38.044112 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032301 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:05:38.044112 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032304 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:05:38.044112 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032307 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:05:38.044112 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032310 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:05:38.044112 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032312 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:05:38.044112 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032315 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:05:38.044112 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032318 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:05:38.044112 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032321 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:05:38.044112 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032323 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:05:38.044112 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032326 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:05:38.044112 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032328 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:05:38.044562 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032331 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:05:38.044562 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032334 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:05:38.044562 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032336 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:05:38.044562 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032339 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:05:38.044562 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032341 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:05:38.044562 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032344 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:05:38.044562 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032346 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:05:38.044562 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032349 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:05:38.044562 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032351 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:05:38.044562 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032354 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:05:38.044562 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032356 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:05:38.044562 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032359 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:05:38.044562 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032362 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:05:38.044562 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032364 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:05:38.044562 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032368 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:05:38.044562 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032370 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:05:38.044562 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032373 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:05:38.044562 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032375 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:05:38.044562 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032378 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:05:38.045620 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032380 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:05:38.045620 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032383 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:05:38.045620 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032385 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:05:38.045620 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032388 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:05:38.045620 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032390 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:05:38.045620 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032393 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:05:38.045620 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032395 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:05:38.045620 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.032398 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:05:38.045620 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.041051 2570 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 23:05:38.045620 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.041071 2570 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 23:05:38.045620 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.041701 2570 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 23:05:38.045620 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.041707 2570 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 23:05:38.045620 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041769 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:05:38.045620 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041775 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:05:38.045620 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041779 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:05:38.045620 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041783 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:05:38.045620 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041786 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:05:38.045620 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041789 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:05:38.045620 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041791 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:05:38.046079 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041794 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:05:38.046079 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041797 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:05:38.046079 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041800 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:05:38.046079 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041803 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:05:38.046079 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041805 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:05:38.046079 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041808 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:05:38.046079 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041811 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:05:38.046079 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041813 2570 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:05:38.046079 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041816 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:05:38.046079 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041818 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:05:38.046079 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041821 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:05:38.046079 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041824 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:05:38.046079 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041826 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:05:38.046079 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041830 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:05:38.046079 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041832 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:05:38.046079 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041835 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:05:38.046079 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041837 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:05:38.046079 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041840 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:05:38.046079 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041843 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:05:38.046528 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041845 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:05:38.046528 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041849 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:05:38.046528 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041851 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:05:38.046528 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041855 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:05:38.046528 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041858 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:05:38.046528 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041860 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:05:38.046528 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041863 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:05:38.046528 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041866 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:05:38.046528 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041868 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:05:38.046528 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041871 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:05:38.046528 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041875 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:05:38.046528 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041879 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:05:38.046528 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041882 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:05:38.046528 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041884 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:05:38.046528 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041887 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:05:38.046528 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041890 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:05:38.046528 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041892 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:05:38.046528 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041895 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:05:38.046528 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041898 2570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:05:38.046967 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041900 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:05:38.046967 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041903 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:05:38.046967 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041906 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:05:38.046967 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041908 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:05:38.046967 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041911 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:05:38.046967 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041913 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:05:38.046967 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041916 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:05:38.046967 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041919 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:05:38.046967 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041921 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:05:38.046967 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041924 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:05:38.046967 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041926 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:05:38.046967 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041929 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:05:38.046967 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041931 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:05:38.046967 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041934 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:05:38.046967 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041937 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:05:38.046967 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041939 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:05:38.046967 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041942 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:05:38.046967 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041945 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:05:38.046967 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041947 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:05:38.047524 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041950 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:05:38.047524 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041952 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:05:38.047524 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041955 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:05:38.047524 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041957 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:05:38.047524 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041960 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:05:38.047524 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041962 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:05:38.047524 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041965 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:05:38.047524 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041967 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:05:38.047524 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041970 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:05:38.047524 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041972 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:05:38.047524 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041975 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:05:38.047524 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041977 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:05:38.047524 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041980 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:05:38.047524 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041982 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:05:38.047524 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041985 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:05:38.047524 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041988 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:05:38.047524 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041991 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:05:38.047524 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041994 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:05:38.047524 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041996 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:05:38.047524 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.041999 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:05:38.047993 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042002 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:05:38.047993 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042004 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:05:38.047993 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042007 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:05:38.047993 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042010 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:05:38.047993 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042012 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:05:38.047993 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042015 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:05:38.047993 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042017 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:05:38.047993 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042020 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:05:38.047993 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042023 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:05:38.047993 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042026 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:05:38.047993 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042028 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:05:38.047993 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042049 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:05:38.047993 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042056 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:05:38.047993 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042063 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:05:38.047993 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042067 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:05:38.047993 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042069 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:05:38.047993 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042072 2570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:05:38.047993 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042075 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:05:38.047993 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042077 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:05:38.048457 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042080 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:05:38.048457 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042082 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:05:38.048457 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042085 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:05:38.048457 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.042113 2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:05:38.048457 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042257 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:05:38.048457 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042262 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:05:38.048457 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042265 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:05:38.048457 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042268 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:05:38.048457 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042271 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:05:38.048457 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042276 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:05:38.048457 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042279 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:05:38.048457 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042281 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:05:38.048457 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042284 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:05:38.048457 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042288 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:05:38.048457 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042292 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:05:38.048457 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042295 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:05:38.048830 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042298 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:05:38.048830 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042301 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:05:38.048830 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042303 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:05:38.048830 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042306 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:05:38.048830 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042308 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:05:38.048830 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042311 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:05:38.048830 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042314 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:05:38.048830 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042317 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:05:38.048830 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042320 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:05:38.048830 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042322 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:05:38.048830 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042325 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:05:38.048830 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042327 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:05:38.048830 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042330 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:05:38.048830 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042332 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:05:38.048830 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042335 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:05:38.048830 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042337 2570 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:05:38.048830 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042340 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:05:38.048830 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042342 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:05:38.048830 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042345 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:05:38.049279 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042347 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:05:38.049279 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042350 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:05:38.049279 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042353 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:05:38.049279 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042355 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:05:38.049279 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042357 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:05:38.049279 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042360 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:05:38.049279 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042363 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:05:38.049279 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042366 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:05:38.049279 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042369 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:05:38.049279 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042371 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:05:38.049279 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042374 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:05:38.049279 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042377 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:05:38.049279 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042379 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:05:38.049279 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042382 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:05:38.049279 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042384 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:05:38.049279 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042387 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:05:38.049279 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042390 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:05:38.049279 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042392 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:05:38.049279 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042395 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:05:38.049721 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042397 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:05:38.049721 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042400 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:05:38.049721 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042402 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:05:38.049721 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042405 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:05:38.049721 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042407 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:05:38.049721 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042410 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:05:38.049721 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042412 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:05:38.049721 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042415 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:05:38.049721 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042417 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:05:38.049721 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042420 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:05:38.049721 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042422 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:05:38.049721 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042425 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:05:38.049721 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042427 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:05:38.049721 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042431 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:05:38.049721 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042435 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:05:38.049721 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042437 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:05:38.049721 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042440 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:05:38.049721 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042442 2570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:05:38.049721 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042445 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:05:38.049721 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042447 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:05:38.050201 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042450 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:05:38.050201 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042453 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:05:38.050201 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042455 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:05:38.050201 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042458 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:05:38.050201 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042460 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:05:38.050201 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042463 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:05:38.050201 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042466 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:05:38.050201 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042469 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:05:38.050201 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042471 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:05:38.050201 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042474 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:05:38.050201 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042476 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:05:38.050201 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042479 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:05:38.050201 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042482 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:05:38.050201 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042484 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:05:38.050201 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042487 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:05:38.050201 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042489 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:05:38.050201 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042491 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:05:38.050201 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042494 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:05:38.050201 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042496 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:05:38.050636 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042499 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:05:38.050636 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042501 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:05:38.050636 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042504 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:05:38.050636 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042506 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:05:38.050636 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042509 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:05:38.050636 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042511 2570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:05:38.050636 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042514 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:05:38.050636 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042516 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:05:38.050636 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042519 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:05:38.050636 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042521 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:05:38.050636 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042524 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:05:38.050636 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042526 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:05:38.050636 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042529 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:05:38.050636 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042532 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:05:38.050636 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042534 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:05:38.050636 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042537 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:05:38.050636 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042540 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:05:38.051031 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.042563 2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:05:38.051031 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042594 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:05:38.051031 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042598 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:05:38.051031 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042601 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:05:38.051031 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042604 2570 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:05:38.051031 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042607 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:05:38.051031 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042610 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:05:38.051031 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042612 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:05:38.051031 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042615 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:05:38.051031 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042617 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:05:38.051031 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042620 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:05:38.051031 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042623 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:05:38.051031 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042625 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:05:38.051031 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042627 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:05:38.051031 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042630 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:05:38.051440 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042632 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:05:38.051440 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042635 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:05:38.051440 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042637 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:05:38.051440 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042640 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:05:38.051440 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042643 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:05:38.051440 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042645 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:05:38.051440 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042647 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:05:38.051440 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042650 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:05:38.051440 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042652 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:05:38.051440 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042655 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:05:38.051440 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042657 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:05:38.051440 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042660 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:05:38.051440 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042662 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:05:38.051440 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042665 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:05:38.051440 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042667 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:05:38.051440 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042670 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:05:38.051440 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042674 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:05:38.051440 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042676 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:05:38.051440 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042679 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:05:38.051440 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042681 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:05:38.051919 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042684 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:05:38.051919 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042686 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:05:38.051919 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042688 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:05:38.051919 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042691 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:05:38.051919 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042693 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:05:38.051919 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042697 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:05:38.051919 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042701 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:05:38.051919 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042704 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:05:38.051919 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042706 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:05:38.051919 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042709 2570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:05:38.051919 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042711 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:05:38.051919 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042714 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:05:38.051919 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042716 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:05:38.051919 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042719 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:05:38.051919 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042721 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:05:38.051919 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042724 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:05:38.051919 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042726 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:05:38.051919 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042729 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:05:38.051919 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042731 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:05:38.051919 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042734 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:05:38.052398 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042737 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:05:38.052398 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042739 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:05:38.052398 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042742 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:05:38.052398 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042745 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:05:38.052398 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042747 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:05:38.052398 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042750 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:05:38.052398 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042752 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:05:38.052398 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042755 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:05:38.052398 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042757 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:05:38.052398 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042760 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:05:38.052398 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042763 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:05:38.052398 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042765 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:05:38.052398 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042768 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:05:38.052398 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042770 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:05:38.052398 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042773 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:05:38.052398 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042775 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:05:38.052398 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042778 2570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:05:38.052398 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042780 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:05:38.052398 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042782 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:05:38.052840 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042785 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:05:38.052840 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042788 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:05:38.052840 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042790 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:05:38.052840 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042793 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:05:38.052840 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042795 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:05:38.052840 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042798 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:05:38.052840 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042800 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:05:38.052840 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042803 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:05:38.052840 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042806 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:05:38.052840 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042808 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:05:38.052840 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042810 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:05:38.052840 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042813 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:05:38.052840 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042815 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:05:38.052840 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042818 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:05:38.052840 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042821 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:05:38.052840 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042823 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:05:38.052840 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042826 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:05:38.052840 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042828 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:05:38.052840 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042832 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:05:38.053294 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042835 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:05:38.053294 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042837 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:05:38.053294 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042840 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:05:38.053294 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042842 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:05:38.053294 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042845 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:05:38.053294 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042848 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:05:38.053294 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042850 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:05:38.053294 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042853 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:05:38.053294 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042855 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:05:38.053294 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042859 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:05:38.053294 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042861 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:05:38.053294 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042864 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:05:38.053294 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042866 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:05:38.053294 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:38.042869 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:05:38.053294 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.042924 2570 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 23:05:38.053294 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.044844 2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 23:05:38.053294 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.045640 2570 server.go:1014] "Starting client certificate rotation" Oct 30 23:05:38.053294 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.045731 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 23:05:38.053294 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.046302 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 23:05:38.053757 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.047603 2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 23:05:38.053757 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.048680 2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 23:05:38.061218 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.061196 2570 log.go:25] "Validated CRI v1 runtime API" Oct 30 23:05:38.063925 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.063908 2570 log.go:25] "Validated CRI v1 image API" Oct 30 23:05:38.064716 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.064687 2570 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 23:05:38.067237 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.067219 2570 fs.go:135] Filesystem UUIDs: map[0e92ea7b-a59c-49db-9b10-7675b5623f00:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c20af26b-c93f-486e-bf9e-32c773c4c313:/dev/nvme0n1p3] Oct 30 23:05:38.067285 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.067237 2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 23:05:38.073326 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.073096 2570 manager.go:217] Machine: {Timestamp:2026-10-30 23:05:38.071493008 +0000 UTC m=+0.306917792 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3090116 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cac8ecdb555cf4ae7cde59fb5c549 SystemUUID:ec2cac8e-cdb5-55cf-4ae7-cde59fb5c549 BootID:551ed6a1-ea6a-4c06-a2a8-3baec0ac93c0 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:02:3e:65:af:c9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:02:3e:65:af:c9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4a:e2:3d:4e:d0:a7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 23:05:38.073862 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.073850 2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 23:05:38.073958 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.073940 2570 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 23:05:38.076926 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.076902 2570 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 23:05:38.077074 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.076929 2570 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-177.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 23:05:38.077115 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.077076 2570 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 23:05:38.077115 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.077084 2570 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 23:05:38.077170 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.077131 2570 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 23:05:38.077170 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.077149 2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 23:05:38.077806 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.077797 2570 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 23:05:38.081585 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.081573 2570 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 23:05:38.081691 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.081682 2570 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 23:05:38.082180 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.082172 2570 kubelet.go:493] "Attempting to sync node with API server" Oct 30 23:05:38.082215 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.082185 2570 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 23:05:38.082215 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.082195 2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 23:05:38.082215 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.082203 2570 kubelet.go:405] "Adding apiserver pod source" Oct 30 23:05:38.082215 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.082211 2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 23:05:38.082510 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.082499 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 23:05:38.084459 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.084432 2570 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 23:05:38.094454 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.094439 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 23:05:38.095336 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.095322 2570 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 23:05:38.095415 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.095371 2570 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 23:05:38.095563 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.095543 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 23:05:38.095626 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.095572 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 23:05:38.095626 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.095583 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 23:05:38.095626 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.095593 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 23:05:38.095626 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.095602 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 23:05:38.095626 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.095610 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 23:05:38.096075 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.096065 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 23:05:38.096133 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.096078 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 23:05:38.096133 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.096089 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 23:05:38.096133 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.096100 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 23:05:38.096133 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.096114 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 23:05:38.096133 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.096128 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 23:05:38.097182 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.097170 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 23:05:38.097249 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.097186 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 23:05:38.100832 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.100816 2570 server.go:1263] "Started kubelet" Oct 30 23:05:38.100931 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.100884 2570 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 23:05:38.101067 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.100942 2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 23:05:38.101114 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.101104 2570 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 23:05:38.101578 ip-10-0-130-177 systemd[1]: Started Kubernetes Kubelet. Oct 30 23:05:38.102271 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.102243 2570 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 23:05:38.103052 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.103019 2570 server.go:317] "Adding debug handlers to kubelet server" Oct 30 23:05:38.106867 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.106850 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 23:05:38.107350 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:38.107311 2570 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 23:05:38.107742 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.107728 2570 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 23:05:38.107830 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:38.107816 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-177.us-west-1.compute.internal\" not found" Oct 30 23:05:38.108413 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.108395 2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 23:05:38.108413 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.108398 2570 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 23:05:38.108547 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.108436 2570 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 23:05:38.108547 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.108504 2570 reconstruct.go:97] "Volume reconstruction finished" Oct 30 23:05:38.108547 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.108515 2570 reconciler.go:29] "Reconciler: start to sync state" Oct 30 23:05:38.108547 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.108507 2570 factory.go:55] Registering systemd factory Oct 30 23:05:38.108547 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.108536 2570 factory.go:223] Registration of the systemd container factory successfully Oct 30 23:05:38.108745 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.108733 2570 factory.go:153] Registering CRI-O factory Oct 30 23:05:38.108778 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.108748 2570 factory.go:223] Registration of the crio container factory successfully Oct 30 23:05:38.108814 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.108805 2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 23:05:38.108848 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.108821 2570 factory.go:103] Registering Raw factory Oct 30 23:05:38.108848 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.108831 2570 manager.go:1199] Started watching for new ooms in manager Oct 30 23:05:38.110929 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.110913 2570 manager.go:319] Starting recovery of all containers Oct 30 23:05:38.121635 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.121617 2570 manager.go:324] Recovery completed Oct 30 23:05:38.125527 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.125514 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:05:38.127579 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.127552 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:05:38.127644 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.127592 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:05:38.127644 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.127602 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 23:05:38.128124 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.128109 2570 cpu_manager.go:226] "Starting" policy="none" Oct 30 23:05:38.128165 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.128124 2570 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 23:05:38.128165 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.128141 2570 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 23:05:38.128223 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.128112 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:05:38.129071 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.129054 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:05:38.129133 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.129087 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:05:38.129133 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.129104 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 23:05:38.130406 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.130394 2570 policy_none.go:50] "Start" Oct 30 23:05:38.130448 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.130410 2570 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 23:05:38.130448 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.130421 2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 23:05:38.132621 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.132609 2570 policy_none.go:44] "Start" Oct 30 23:05:38.190526 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.170746 2570 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 23:05:38.190526 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.176298 2570 manager.go:341] "Starting Device Plugin manager" Oct 30 23:05:38.190526 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:38.176322 2570 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 23:05:38.190526 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.176331 2570 server.go:87] "Starting device plugin registration server" Oct 30 23:05:38.190526 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.176541 2570 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 23:05:38.190526 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.176550 2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 23:05:38.190526 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.176693 2570 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 23:05:38.190526 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.176807 2570 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 23:05:38.190526 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.176816 2570 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 23:05:38.190526 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:38.178153 2570 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 23:05:38.190526 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:38.178206 2570 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-177.us-west-1.compute.internal\" not found" Oct 30 23:05:38.190526 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.182776 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 23:05:38.190526 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.184133 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 23:05:38.190526 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.184150 2570 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 23:05:38.190526 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.184169 2570 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 23:05:38.190526 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:38.184197 2570 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 23:05:38.241111 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.241086 2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-177.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 23:05:38.241217 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:38.241112 2570 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-177.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 23:05:38.242165 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:38.241096 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-177.us-west-1.compute.internal.18e371327223052f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-177.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-177.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 23:05:38.100790575 +0000 UTC m=+0.336215355,LastTimestamp:2026-10-30 23:05:38.100790575 +0000 UTC m=+0.336215355,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-177.us-west-1.compute.internal,}" Oct 30 23:05:38.277173 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.277127 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:05:38.277871 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.277849 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:05:38.277962 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.277881 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:05:38.277962 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.277893 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 23:05:38.277962 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.277915 2570 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:05:38.284503 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.284485 2570 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-177.us-west-1.compute.internal"] Oct 30 23:05:38.284575 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.284539 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:05:38.285255 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.285239 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:05:38.285332 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.285271 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:05:38.285332 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.285294 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 23:05:38.287600 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.287586 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:05:38.287732 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.287714 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:05:38.287778 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.287749 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:05:38.288272 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.288257 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:05:38.288272 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.288266 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:05:38.288382 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.288288 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:05:38.288382 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.288289 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:05:38.288382 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.288298 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 23:05:38.288382 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.288304 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 23:05:38.290303 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.290289 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:05:38.290389 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.290319 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:05:38.290999 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.290985 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:05:38.291093 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.291011 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:05:38.291093 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.291026 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 23:05:38.309265 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:38.309248 2570 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-177.us-west-1.compute.internal\" not found" node="ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:05:38.310411 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.310394 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5d800e6e610855f53a5c54432ef51ba0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal\" (UID: \"5d800e6e610855f53a5c54432ef51ba0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:05:38.310483 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.310426 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d800e6e610855f53a5c54432ef51ba0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal\" (UID: \"5d800e6e610855f53a5c54432ef51ba0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:05:38.310483 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.310452 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f343176fefb75130c5aad85365de861f-config\") pod \"kube-apiserver-proxy-ip-10-0-130-177.us-west-1.compute.internal\" (UID: \"f343176fefb75130c5aad85365de861f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:05:38.311286 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:38.311269 2570 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-177.us-west-1.compute.internal\" not found" node="ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:05:38.318451 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.318435 2570 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 23:05:38.318529 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.318498 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kj7nv" Oct 30 23:05:38.348700 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:38.348674 2570 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 23:05:38.348800 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:38.348676 2570 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 23:05:38.348909 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:38.348832 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-177.us-west-1.compute.internal.18e3713273bbc178 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-177.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-177.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-177.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 23:05:38.127577464 +0000 UTC m=+0.363002248,LastTimestamp:2026-10-30 23:05:38.127577464 +0000 UTC m=+0.363002248,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-177.us-west-1.compute.internal,}" Oct 30 23:05:38.399098 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:38.399074 2570 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Oct 30 23:05:38.399179 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:38.399103 2570 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-177.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 23:05:38.400450 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:38.400435 2570 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-130-177.us-west-1.compute.internal\" not found" node="ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:05:38.411514 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.411495 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5d800e6e610855f53a5c54432ef51ba0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal\" (UID: \"5d800e6e610855f53a5c54432ef51ba0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:05:38.411593 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.411525 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d800e6e610855f53a5c54432ef51ba0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal\" (UID: \"5d800e6e610855f53a5c54432ef51ba0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:05:38.411593 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.411552 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f343176fefb75130c5aad85365de861f-config\") pod \"kube-apiserver-proxy-ip-10-0-130-177.us-west-1.compute.internal\" (UID: \"f343176fefb75130c5aad85365de861f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:05:38.411688 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.411591 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5d800e6e610855f53a5c54432ef51ba0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal\" (UID: \"5d800e6e610855f53a5c54432ef51ba0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:05:38.411688 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.411601 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f343176fefb75130c5aad85365de861f-config\") pod \"kube-apiserver-proxy-ip-10-0-130-177.us-west-1.compute.internal\" (UID: \"f343176fefb75130c5aad85365de861f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:05:38.411688 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.411604 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d800e6e610855f53a5c54432ef51ba0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal\" (UID: \"5d800e6e610855f53a5c54432ef51ba0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:05:38.544335 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:38.544282 2570 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-130-177.us-west-1.compute.internal\" not found" node="ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:05:38.600678 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.600662 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:05:38.601568 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.601548 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:05:38.601636 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.601580 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:05:38.601636 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.601594 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 23:05:38.603307 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.601924 2570 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:05:38.611857 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.611837 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:05:38.614886 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.614870 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:05:38.661904 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:38.661886 2570 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:05:38.661904 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:38.661904 2570 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-177.us-west-1.compute.internal\": node \"ip-10-0-130-177.us-west-1.compute.internal\" not found" Oct 30 23:05:38.782067 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:38.782030 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-177.us-west-1.compute.internal\" not found" Oct 30 23:05:38.883010 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:38.882958 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-177.us-west-1.compute.internal\" not found" Oct 30 23:05:38.983085 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:38.983062 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-177.us-west-1.compute.internal\" not found" Oct 30 23:05:39.046457 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:39.046430 2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 23:05:39.046833 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:39.046550 2570 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a69c238be523043a7aa0939d4b6f2b19-5c317881c9fa0372.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-130-177.us-west-1.compute.internal.18e3713273bc0de8\": read tcp 10.0.130.177:56066->3.16.240.75:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-130-177.us-west-1.compute.internal.18e3713273bc0de8 default 4709 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-177.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-177.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-177.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 23:05:38 +0000 UTC,LastTimestamp:2026-10-30 23:05:38.285286051 +0000 UTC m=+0.520710836,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-177.us-west-1.compute.internal,}" Oct 30 23:05:39.083811 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:39.083791 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-177.us-west-1.compute.internal\" not found" Oct 30 23:05:39.184500 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:39.184437 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-177.us-west-1.compute.internal\" not found" Oct 30 23:05:39.256743 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:39.256722 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:05:39.257551 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:39.257535 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:05:39.257631 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:39.257564 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:05:39.257631 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:39.257576 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 23:05:39.285118 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:39.285098 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-177.us-west-1.compute.internal\" not found" Oct 30 23:05:39.323406 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:39.323380 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 23:00:38 +0000 UTC" deadline="2028-07-31 02:28:10.787157582 +0000 UTC" Oct 30 23:05:39.323406 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:39.323402 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15339h22m31.463758356s" Oct 30 23:05:39.360313 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:39.360290 2570 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:05:39.360398 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:39.360384 2570 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:05:39.408684 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:39.408664 2570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:05:39.466620 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:39.466569 2570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:05:39.468783 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:39.468770 2570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:05:39.559359 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:39.559332 2570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:05:39.559467 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:39.559453 2570 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:05:39.710680 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:39.710644 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf343176fefb75130c5aad85365de861f.slice/crio-dd4fa69b3d38d1465c0b30f5f395947b0ff1d833244a71f20b8924d08ad75ed9 WatchSource:0}: Error finding container dd4fa69b3d38d1465c0b30f5f395947b0ff1d833244a71f20b8924d08ad75ed9: Status 404 returned error can't find the container with id dd4fa69b3d38d1465c0b30f5f395947b0ff1d833244a71f20b8924d08ad75ed9 Oct 30 23:05:39.711071 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:39.711031 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d800e6e610855f53a5c54432ef51ba0.slice/crio-d57a493fc14ecfecab1a187940f32a6090599edd683d5d078a4f75bb8e8846ad WatchSource:0}: Error finding container d57a493fc14ecfecab1a187940f32a6090599edd683d5d078a4f75bb8e8846ad: Status 404 returned error can't find the container with id d57a493fc14ecfecab1a187940f32a6090599edd683d5d078a4f75bb8e8846ad Oct 30 23:05:39.716093 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:39.716077 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 23:05:39.876501 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:39.876477 2570 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:05:40.082941 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.082872 2570 apiserver.go:52] "Watching apiserver" Oct 30 23:05:40.108354 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.108332 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 23:05:40.188410 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.188365 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-177.us-west-1.compute.internal" event={"ID":"f343176fefb75130c5aad85365de861f","Type":"ContainerStarted","Data":"dd4fa69b3d38d1465c0b30f5f395947b0ff1d833244a71f20b8924d08ad75ed9"} Oct 30 23:05:40.189602 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.189581 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal" event={"ID":"5d800e6e610855f53a5c54432ef51ba0","Type":"ContainerStarted","Data":"d57a493fc14ecfecab1a187940f32a6090599edd683d5d078a4f75bb8e8846ad"} Oct 30 23:05:40.190675 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.190660 2570 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 23:05:40.190948 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.190933 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-74qh7","openshift-multus/multus-zx7t9","openshift-ovn-kubernetes/ovnkube-node-ts59n","kube-system/konnectivity-agent-9sp4c","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-9cltq","openshift-multus/network-metrics-daemon-htqtp","openshift-network-operator/iptables-alerter-khr89","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz","openshift-dns/node-resolver-p84ng","openshift-network-diagnostics/network-check-target-xdlsv","kube-system/kube-apiserver-proxy-ip-10-0-130-177.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-z4l28","kube-system/global-pull-secret-syncer-7vrpr"] Oct 30 23:05:40.194973 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.194953 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:05:40.195089 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:40.195069 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-htqtp" podUID="dec2c5fa-4b9a-47ed-b005-04874761f419" Oct 30 23:05:40.197080 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.197059 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdlsv" Oct 30 23:05:40.197158 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:40.197132 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xdlsv" podUID="4f7d22b5-256e-47d8-947f-b89b47f14d08" Oct 30 23:05:40.199477 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.199463 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.203460 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.203440 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9sp4c" Oct 30 23:05:40.205707 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.205688 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:05:40.205784 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:40.205760 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vrpr" podUID="322e3a9b-911f-4cd8-8f43-71d56152e917" Oct 30 23:05:40.208119 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.208093 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:05:40.210423 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.210405 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:05:40.213144 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.212916 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.215290 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.215275 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.217585 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.217570 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p84ng" Oct 30 23:05:40.220052 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.219913 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-74qh7" Oct 30 23:05:40.222356 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.222336 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-khr89" Oct 30 23:05:40.222623 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.222605 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-run-openvswitch\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.222697 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.222636 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e4d82728-b04b-4914-93d2-fce161acae36-agent-certs\") pod \"konnectivity-agent-9sp4c\" (UID: \"e4d82728-b04b-4914-93d2-fce161acae36\") " pod="kube-system/konnectivity-agent-9sp4c" Oct 30 23:05:40.222737 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.222716 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4368706-761a-4f6e-9dda-a8087a5f4fdc-system-cni-dir\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:05:40.222790 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.222770 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4368706-761a-4f6e-9dda-a8087a5f4fdc-cni-binary-copy\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:05:40.222828 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.222817 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c4368706-761a-4f6e-9dda-a8087a5f4fdc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:05:40.222891 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.222861 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-run-netns\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.222997 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.222909 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-env-overrides\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.222997 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.222941 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs\") pod \"network-metrics-daemon-htqtp\" (UID: \"dec2c5fa-4b9a-47ed-b005-04874761f419\") " pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:05:40.222997 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.222987 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-run-ovn\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.223120 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.223029 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-cni-bin\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.223120 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.223078 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-ovn-node-metrics-cert\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.223120 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.223104 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-run-systemd\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.223231 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.223147 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4368706-761a-4f6e-9dda-a8087a5f4fdc-cnibin\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:05:40.223231 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.223172 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-etc-selinux\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:05:40.223231 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.223194 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-log-socket\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.223231 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.223213 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-ovnkube-config\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.223468 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.223248 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-ovnkube-script-lib\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.223468 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.223302 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e4d82728-b04b-4914-93d2-fce161acae36-konnectivity-ca\") pod \"konnectivity-agent-9sp4c\" (UID: \"e4d82728-b04b-4914-93d2-fce161acae36\") " pod="kube-system/konnectivity-agent-9sp4c" Oct 30 23:05:40.223468 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.223347 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9g9l\" (UniqueName: \"kubernetes.io/projected/dec2c5fa-4b9a-47ed-b005-04874761f419-kube-api-access-r9g9l\") pod \"network-metrics-daemon-htqtp\" (UID: \"dec2c5fa-4b9a-47ed-b005-04874761f419\") " pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:05:40.223468 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.223404 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4368706-761a-4f6e-9dda-a8087a5f4fdc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:05:40.223468 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.223434 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-sys-fs\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:05:40.223650 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.223497 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-var-lib-openvswitch\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.223650 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.223529 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-run-ovn-kubernetes\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.223650 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.223555 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.223650 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.223595 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/322e3a9b-911f-4cd8-8f43-71d56152e917-dbus\") pod \"global-pull-secret-syncer-7vrpr\" (UID: \"322e3a9b-911f-4cd8-8f43-71d56152e917\") " pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:05:40.223650 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.223624 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-kubelet-dir\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:05:40.223650 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.223646 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc5qz\" (UniqueName: \"kubernetes.io/projected/2861d402-9b72-4907-b8c1-7199ee3a8e92-kube-api-access-pc5qz\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:05:40.223862 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.223681 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-slash\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.223862 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.223701 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-etc-openvswitch\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.223862 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.223717 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-node-log\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.223862 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.223741 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-cni-netd\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.223862 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.223774 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/322e3a9b-911f-4cd8-8f43-71d56152e917-original-pull-secret\") pod \"global-pull-secret-syncer-7vrpr\" (UID: \"322e3a9b-911f-4cd8-8f43-71d56152e917\") " pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:05:40.223862 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.223803 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-registration-dir\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:05:40.223862 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.223840 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9nfl\" (UniqueName: \"kubernetes.io/projected/4f7d22b5-256e-47d8-947f-b89b47f14d08-kube-api-access-l9nfl\") pod \"network-check-target-xdlsv\" (UID: \"4f7d22b5-256e-47d8-947f-b89b47f14d08\") " pod="openshift-network-diagnostics/network-check-target-xdlsv" Oct 30 23:05:40.223862 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.223860 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtz5v\" (UniqueName: \"kubernetes.io/projected/c4368706-761a-4f6e-9dda-a8087a5f4fdc-kube-api-access-qtz5v\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:05:40.224241 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.223876 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-device-dir\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:05:40.224241 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.223904 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-kubelet\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.224241 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.223947 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-systemd-units\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.224241 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.223988 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4cs6\" (UniqueName: \"kubernetes.io/projected/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-kube-api-access-b4cs6\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.224241 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.224017 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/322e3a9b-911f-4cd8-8f43-71d56152e917-kubelet-config\") pod \"global-pull-secret-syncer-7vrpr\" (UID: \"322e3a9b-911f-4cd8-8f43-71d56152e917\") " pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:05:40.224241 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.224069 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c4368706-761a-4f6e-9dda-a8087a5f4fdc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:05:40.224241 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.224101 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-socket-dir\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:05:40.258752 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.258730 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 23:05:40.259029 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.259013 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 23:05:40.259140 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.259014 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 23:05:40.259345 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.259016 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 23:05:40.259527 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.259501 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 23:05:40.259636 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.259595 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t7twr\"" Oct 30 23:05:40.259694 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.259635 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 23:05:40.260810 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.260791 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 23:05:40.260984 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.260968 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 23:05:40.261047 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.260993 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4qgbv\"" Oct 30 23:05:40.264878 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.264861 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 23:05:40.265290 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.265277 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 23:05:40.265391 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.265312 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 23:05:40.265391 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.265340 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 23:05:40.265500 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.265417 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 23:05:40.265500 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.265432 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7sr97\"" Oct 30 23:05:40.267092 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.267054 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 23:05:40.267210 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.267197 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 23:05:40.267390 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.267375 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-46zpf\"" Oct 30 23:05:40.267450 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.267400 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 23:05:40.270028 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.270013 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:05:40.270097 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.270075 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9xqsx\"" Oct 30 23:05:40.303408 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.303368 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 23:05:40.303527 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.303418 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-7m4ss\"" Oct 30 23:05:40.303527 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.303450 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 23:05:40.303527 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.303452 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 23:05:40.303527 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.303491 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4j2qm\"" Oct 30 23:05:40.303527 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.303506 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:05:40.303527 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.303493 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 23:05:40.303527 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.303523 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 23:05:40.303527 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.303527 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 23:05:40.303816 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.303529 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nlg56\"" Oct 30 23:05:40.303816 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.303552 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ldcvv\"" Oct 30 23:05:40.303816 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.303555 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 23:05:40.303816 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.303577 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 23:05:40.303816 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.303579 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 23:05:40.303816 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.303772 2570 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 23:05:40.303816 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.303814 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-v8rwc" Oct 30 23:05:40.309473 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.309449 2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 23:05:40.324455 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.324431 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-socket-dir\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:05:40.324563 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.324545 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-run-openvswitch\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.324626 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.324574 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-run-openvswitch\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.324626 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.324580 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e4d82728-b04b-4914-93d2-fce161acae36-agent-certs\") pod \"konnectivity-agent-9sp4c\" (UID: \"e4d82728-b04b-4914-93d2-fce161acae36\") " pod="kube-system/konnectivity-agent-9sp4c" Oct 30 23:05:40.324626 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.324543 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-socket-dir\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:05:40.324626 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.324612 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-cnibin\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.324797 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.324638 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-systemd\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.324797 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.324664 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4368706-761a-4f6e-9dda-a8087a5f4fdc-cni-binary-copy\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:05:40.324797 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.324751 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-env-overrides\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.324797 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.324777 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-run-ovn\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.324995 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.324797 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-ovn-node-metrics-cert\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.324995 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.324826 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjr2r\" (UniqueName: \"kubernetes.io/projected/3838326a-857d-4511-8866-813015d85554-kube-api-access-rjr2r\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.324995 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.324851 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-multus-cni-dir\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.324995 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.324882 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-run-ovn\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.324995 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.324923 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-multus-socket-dir-parent\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.324995 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.324943 2570 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 23:05:40.324995 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.324954 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-log-socket\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.324995 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.324981 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-ovnkube-config\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.325418 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.325024 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e4d82728-b04b-4914-93d2-fce161acae36-konnectivity-ca\") pod \"konnectivity-agent-9sp4c\" (UID: \"e4d82728-b04b-4914-93d2-fce161acae36\") " pod="kube-system/konnectivity-agent-9sp4c" Oct 30 23:05:40.325418 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.325069 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9g9l\" (UniqueName: \"kubernetes.io/projected/dec2c5fa-4b9a-47ed-b005-04874761f419-kube-api-access-r9g9l\") pod \"network-metrics-daemon-htqtp\" (UID: \"dec2c5fa-4b9a-47ed-b005-04874761f419\") " pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:05:40.325418 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.325100 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-lib-modules\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.325418 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.325124 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3838326a-857d-4511-8866-813015d85554-tmp\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.325418 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.325151 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-sys-fs\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:05:40.325418 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.325176 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-var-lib-openvswitch\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.325418 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.325202 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.325418 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.325232 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-var-lib-cni-bin\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.325418 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.325260 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-var-lib-kubelet\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.325418 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.325284 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1cc88905-e18f-4a2d-a503-4b4950cd27d3-host\") pod \"node-ca-74qh7\" (UID: \"1cc88905-e18f-4a2d-a503-4b4950cd27d3\") " pod="openshift-image-registry/node-ca-74qh7" Oct 30 23:05:40.325418 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.325308 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5cd7decf-5213-4360-a405-76d07c6ae3df-hosts-file\") pod \"node-resolver-p84ng\" (UID: \"5cd7decf-5213-4360-a405-76d07c6ae3df\") " pod="openshift-dns/node-resolver-p84ng" Oct 30 23:05:40.325418 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.325337 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-kubelet-dir\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:05:40.325418 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.325370 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc5qz\" (UniqueName: \"kubernetes.io/projected/2861d402-9b72-4907-b8c1-7199ee3a8e92-kube-api-access-pc5qz\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:05:40.325418 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.325397 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-run-netns\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.325418 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.325423 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-hostroot\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.326195 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.325446 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-multus-conf-dir\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.326195 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.325307 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-log-socket\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.326195 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.325551 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-kubelet-dir\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:05:40.326195 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.325614 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-sys-fs\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:05:40.326195 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.325653 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-var-lib-openvswitch\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.326195 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.325683 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-run-multus-certs\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.326195 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.325820 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.326195 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.325859 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/322e3a9b-911f-4cd8-8f43-71d56152e917-original-pull-secret\") pod \"global-pull-secret-syncer-7vrpr\" (UID: \"322e3a9b-911f-4cd8-8f43-71d56152e917\") " pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:05:40.326195 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.325888 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3838326a-857d-4511-8866-813015d85554-etc-tuned\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.326195 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.325919 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9nfl\" (UniqueName: \"kubernetes.io/projected/4f7d22b5-256e-47d8-947f-b89b47f14d08-kube-api-access-l9nfl\") pod \"network-check-target-xdlsv\" (UID: \"4f7d22b5-256e-47d8-947f-b89b47f14d08\") " pod="openshift-network-diagnostics/network-check-target-xdlsv" Oct 30 23:05:40.326195 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:40.326007 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:05:40.326195 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.326072 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-device-dir\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:05:40.326195 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.326107 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-systemd-units\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.326195 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:40.326146 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/322e3a9b-911f-4cd8-8f43-71d56152e917-original-pull-secret podName:322e3a9b-911f-4cd8-8f43-71d56152e917 nodeName:}" failed. No retries permitted until 2026-10-30 23:05:40.826106425 +0000 UTC m=+3.061531218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/322e3a9b-911f-4cd8-8f43-71d56152e917-original-pull-secret") pod "global-pull-secret-syncer-7vrpr" (UID: "322e3a9b-911f-4cd8-8f43-71d56152e917") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:05:40.326195 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.326154 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-systemd-units\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.326195 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.326174 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4cs6\" (UniqueName: \"kubernetes.io/projected/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-kube-api-access-b4cs6\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.326195 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.326188 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-env-overrides\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.327021 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.326203 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c4368706-761a-4f6e-9dda-a8087a5f4fdc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:05:40.327021 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.326204 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-device-dir\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:05:40.327021 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.326258 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/706016da-3413-43e2-b06e-944572ca4106-host-slash\") pod \"iptables-alerter-khr89\" (UID: \"706016da-3413-43e2-b06e-944572ca4106\") " pod="openshift-network-operator/iptables-alerter-khr89" Oct 30 23:05:40.327021 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.326348 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4368706-761a-4f6e-9dda-a8087a5f4fdc-cni-binary-copy\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:05:40.327021 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.326401 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4368706-761a-4f6e-9dda-a8087a5f4fdc-system-cni-dir\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:05:40.327021 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.326445 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c4368706-761a-4f6e-9dda-a8087a5f4fdc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:05:40.327021 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.326458 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4368706-761a-4f6e-9dda-a8087a5f4fdc-system-cni-dir\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:05:40.327021 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.326483 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-ovnkube-config\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.327021 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.326500 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-run-netns\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.327021 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.326544 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-run-netns\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.327021 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.326542 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs\") pod \"network-metrics-daemon-htqtp\" (UID: \"dec2c5fa-4b9a-47ed-b005-04874761f419\") " pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:05:40.327021 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.326557 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e4d82728-b04b-4914-93d2-fce161acae36-konnectivity-ca\") pod \"konnectivity-agent-9sp4c\" (UID: \"e4d82728-b04b-4914-93d2-fce161acae36\") " pod="kube-system/konnectivity-agent-9sp4c" Oct 30 23:05:40.327021 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:40.326612 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:05:40.327021 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.326647 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-sys\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.327021 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:40.326713 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs podName:dec2c5fa-4b9a-47ed-b005-04874761f419 nodeName:}" failed. No retries permitted until 2026-10-30 23:05:40.826692333 +0000 UTC m=+3.062117118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs") pod "network-metrics-daemon-htqtp" (UID: "dec2c5fa-4b9a-47ed-b005-04874761f419") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:05:40.327021 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.326775 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-host\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.327021 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.326815 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-cni-bin\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.327650 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.326846 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-run-k8s-cni-cncf-io\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.327650 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.326900 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1cc88905-e18f-4a2d-a503-4b4950cd27d3-serviceca\") pod \"node-ca-74qh7\" (UID: \"1cc88905-e18f-4a2d-a503-4b4950cd27d3\") " pod="openshift-image-registry/node-ca-74qh7" Oct 30 23:05:40.327650 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.326920 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c4368706-761a-4f6e-9dda-a8087a5f4fdc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:05:40.327650 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.326932 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-sysconfig\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.327650 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.326961 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tm5m\" (UniqueName: \"kubernetes.io/projected/5cd7decf-5213-4360-a405-76d07c6ae3df-kube-api-access-9tm5m\") pod \"node-resolver-p84ng\" (UID: \"5cd7decf-5213-4360-a405-76d07c6ae3df\") " pod="openshift-dns/node-resolver-p84ng" Oct 30 23:05:40.327650 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.326966 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-cni-bin\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.327650 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.326995 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-system-cni-dir\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.327650 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327028 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-var-lib-cni-multus\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.327650 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327047 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-run-systemd\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.327650 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327082 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4368706-761a-4f6e-9dda-a8087a5f4fdc-cnibin\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:05:40.327650 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327100 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-etc-selinux\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:05:40.327650 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327119 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-ovnkube-script-lib\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.327650 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327133 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-run-systemd\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.327650 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327124 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c4368706-761a-4f6e-9dda-a8087a5f4fdc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:05:40.327650 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327149 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q42l2\" (UniqueName: \"kubernetes.io/projected/1cc88905-e18f-4a2d-a503-4b4950cd27d3-kube-api-access-q42l2\") pod \"node-ca-74qh7\" (UID: \"1cc88905-e18f-4a2d-a503-4b4950cd27d3\") " pod="openshift-image-registry/node-ca-74qh7" Oct 30 23:05:40.327650 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327178 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4368706-761a-4f6e-9dda-a8087a5f4fdc-cnibin\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:05:40.327650 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327192 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4368706-761a-4f6e-9dda-a8087a5f4fdc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:05:40.328260 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327269 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-run-ovn-kubernetes\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.328260 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327303 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-etc-selinux\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:05:40.328260 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327310 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5bfb1046-f8b5-4035-bb35-173a1316ea0f-multus-daemon-config\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.328260 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327339 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4368706-761a-4f6e-9dda-a8087a5f4fdc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:05:40.328260 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327361 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-run-ovn-kubernetes\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.328260 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327408 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-etc-kubernetes\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.328260 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327451 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lftl4\" (UniqueName: \"kubernetes.io/projected/5bfb1046-f8b5-4035-bb35-173a1316ea0f-kube-api-access-lftl4\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.328260 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327484 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t9v8\" (UniqueName: \"kubernetes.io/projected/706016da-3413-43e2-b06e-944572ca4106-kube-api-access-7t9v8\") pod \"iptables-alerter-khr89\" (UID: \"706016da-3413-43e2-b06e-944572ca4106\") " pod="openshift-network-operator/iptables-alerter-khr89" Oct 30 23:05:40.328260 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327525 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-var-lib-kubelet\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.328260 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327561 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/322e3a9b-911f-4cd8-8f43-71d56152e917-dbus\") pod \"global-pull-secret-syncer-7vrpr\" (UID: \"322e3a9b-911f-4cd8-8f43-71d56152e917\") " pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:05:40.328260 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327584 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-slash\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.328260 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327619 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-slash\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.328260 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327642 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-etc-openvswitch\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.328260 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327663 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-node-log\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.328260 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327680 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-cni-netd\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.328260 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327699 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/322e3a9b-911f-4cd8-8f43-71d56152e917-dbus\") pod \"global-pull-secret-syncer-7vrpr\" (UID: \"322e3a9b-911f-4cd8-8f43-71d56152e917\") " pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:05:40.328260 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327721 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-ovnkube-script-lib\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.328781 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327731 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-etc-openvswitch\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.328781 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327762 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-cni-netd\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.328781 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327779 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-node-log\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.328781 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327702 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-modprobe-d\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.328781 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327826 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-sysctl-d\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.328781 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327851 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-run\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.328781 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327879 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-registration-dir\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:05:40.328781 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327903 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bfb1046-f8b5-4035-bb35-173a1316ea0f-cni-binary-copy\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.328781 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.327955 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-registration-dir\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:05:40.328781 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.328014 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/706016da-3413-43e2-b06e-944572ca4106-iptables-alerter-script\") pod \"iptables-alerter-khr89\" (UID: \"706016da-3413-43e2-b06e-944572ca4106\") " pod="openshift-network-operator/iptables-alerter-khr89" Oct 30 23:05:40.328781 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.328065 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cd7decf-5213-4360-a405-76d07c6ae3df-tmp-dir\") pod \"node-resolver-p84ng\" (UID: \"5cd7decf-5213-4360-a405-76d07c6ae3df\") " pod="openshift-dns/node-resolver-p84ng" Oct 30 23:05:40.328781 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.328108 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtz5v\" (UniqueName: \"kubernetes.io/projected/c4368706-761a-4f6e-9dda-a8087a5f4fdc-kube-api-access-qtz5v\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:05:40.328781 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.328152 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-kubelet\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.328781 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.328179 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-kubernetes\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.328781 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.328207 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-sysctl-conf\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.328781 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.328257 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/322e3a9b-911f-4cd8-8f43-71d56152e917-kubelet-config\") pod \"global-pull-secret-syncer-7vrpr\" (UID: \"322e3a9b-911f-4cd8-8f43-71d56152e917\") " pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:05:40.328781 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.328346 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/322e3a9b-911f-4cd8-8f43-71d56152e917-kubelet-config\") pod \"global-pull-secret-syncer-7vrpr\" (UID: \"322e3a9b-911f-4cd8-8f43-71d56152e917\") " pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:05:40.329337 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.328603 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-kubelet\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.329337 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.328907 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-ovn-node-metrics-cert\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.329337 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.329081 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e4d82728-b04b-4914-93d2-fce161acae36-agent-certs\") pod \"konnectivity-agent-9sp4c\" (UID: \"e4d82728-b04b-4914-93d2-fce161acae36\") " pod="kube-system/konnectivity-agent-9sp4c" Oct 30 23:05:40.386560 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:40.386505 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:05:40.386560 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:40.386528 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:05:40.386560 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:40.386541 2570 projected.go:196] Error preparing data for projected volume kube-api-access-l9nfl for pod openshift-network-diagnostics/network-check-target-xdlsv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:05:40.386921 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:40.386592 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f7d22b5-256e-47d8-947f-b89b47f14d08-kube-api-access-l9nfl podName:4f7d22b5-256e-47d8-947f-b89b47f14d08 nodeName:}" failed. No retries permitted until 2026-10-30 23:05:40.886576196 +0000 UTC m=+3.122000988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-l9nfl" (UniqueName: "kubernetes.io/projected/4f7d22b5-256e-47d8-947f-b89b47f14d08-kube-api-access-l9nfl") pod "network-check-target-xdlsv" (UID: "4f7d22b5-256e-47d8-947f-b89b47f14d08") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:05:40.388683 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.388598 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4cs6\" (UniqueName: \"kubernetes.io/projected/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-kube-api-access-b4cs6\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.388683 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.388642 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc5qz\" (UniqueName: \"kubernetes.io/projected/2861d402-9b72-4907-b8c1-7199ee3a8e92-kube-api-access-pc5qz\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:05:40.388872 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.388772 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9g9l\" (UniqueName: \"kubernetes.io/projected/dec2c5fa-4b9a-47ed-b005-04874761f419-kube-api-access-r9g9l\") pod \"network-metrics-daemon-htqtp\" (UID: \"dec2c5fa-4b9a-47ed-b005-04874761f419\") " pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:05:40.390213 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.390193 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtz5v\" (UniqueName: \"kubernetes.io/projected/c4368706-761a-4f6e-9dda-a8087a5f4fdc-kube-api-access-qtz5v\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:05:40.428711 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.428683 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-cnibin\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.428807 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.428713 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-systemd\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.428807 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.428732 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjr2r\" (UniqueName: \"kubernetes.io/projected/3838326a-857d-4511-8866-813015d85554-kube-api-access-rjr2r\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.428807 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.428747 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-multus-cni-dir\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.428807 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.428761 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-multus-socket-dir-parent\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.428807 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.428778 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-lib-modules\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.428807 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.428794 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3838326a-857d-4511-8866-813015d85554-tmp\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.429158 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.428813 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-var-lib-cni-bin\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.429158 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.428813 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-systemd\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.429158 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.428838 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-var-lib-kubelet\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.429158 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.428836 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-cnibin\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.429158 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.428836 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-multus-cni-dir\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.429158 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.428859 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-multus-socket-dir-parent\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.429158 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.428875 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-var-lib-cni-bin\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.429158 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.428898 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1cc88905-e18f-4a2d-a503-4b4950cd27d3-host\") pod \"node-ca-74qh7\" (UID: \"1cc88905-e18f-4a2d-a503-4b4950cd27d3\") " pod="openshift-image-registry/node-ca-74qh7" Oct 30 23:05:40.429158 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.428908 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-var-lib-kubelet\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.429158 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.428927 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5cd7decf-5213-4360-a405-76d07c6ae3df-hosts-file\") pod \"node-resolver-p84ng\" (UID: \"5cd7decf-5213-4360-a405-76d07c6ae3df\") " pod="openshift-dns/node-resolver-p84ng" Oct 30 23:05:40.429158 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.428932 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-lib-modules\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.429158 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.428934 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1cc88905-e18f-4a2d-a503-4b4950cd27d3-host\") pod \"node-ca-74qh7\" (UID: \"1cc88905-e18f-4a2d-a503-4b4950cd27d3\") " pod="openshift-image-registry/node-ca-74qh7" Oct 30 23:05:40.429158 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.428953 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-run-netns\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.429158 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.428976 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-hostroot\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.429158 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.428999 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-multus-conf-dir\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.429158 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429020 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-hostroot\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.429158 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429024 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-run-netns\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.429158 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429037 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-run-multus-certs\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.429158 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429081 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5cd7decf-5213-4360-a405-76d07c6ae3df-hosts-file\") pod \"node-resolver-p84ng\" (UID: \"5cd7decf-5213-4360-a405-76d07c6ae3df\") " pod="openshift-dns/node-resolver-p84ng" Oct 30 23:05:40.430102 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429089 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-multus-conf-dir\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.430102 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429106 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3838326a-857d-4511-8866-813015d85554-etc-tuned\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.430102 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429130 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-run-multus-certs\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.430102 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429136 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/706016da-3413-43e2-b06e-944572ca4106-host-slash\") pod \"iptables-alerter-khr89\" (UID: \"706016da-3413-43e2-b06e-944572ca4106\") " pod="openshift-network-operator/iptables-alerter-khr89" Oct 30 23:05:40.430102 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429177 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-sys\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.430102 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429222 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/706016da-3413-43e2-b06e-944572ca4106-host-slash\") pod \"iptables-alerter-khr89\" (UID: \"706016da-3413-43e2-b06e-944572ca4106\") " pod="openshift-network-operator/iptables-alerter-khr89" Oct 30 23:05:40.430102 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429224 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-sys\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.430102 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429264 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-host\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.430102 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429295 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-run-k8s-cni-cncf-io\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.430102 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429324 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1cc88905-e18f-4a2d-a503-4b4950cd27d3-serviceca\") pod \"node-ca-74qh7\" (UID: \"1cc88905-e18f-4a2d-a503-4b4950cd27d3\") " pod="openshift-image-registry/node-ca-74qh7" Oct 30 23:05:40.430102 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429333 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-host\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.430102 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429352 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-sysconfig\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.430102 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429379 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tm5m\" (UniqueName: \"kubernetes.io/projected/5cd7decf-5213-4360-a405-76d07c6ae3df-kube-api-access-9tm5m\") pod \"node-resolver-p84ng\" (UID: \"5cd7decf-5213-4360-a405-76d07c6ae3df\") " pod="openshift-dns/node-resolver-p84ng" Oct 30 23:05:40.430102 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429405 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-system-cni-dir\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.430102 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429433 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-var-lib-cni-multus\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.430102 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429462 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q42l2\" (UniqueName: \"kubernetes.io/projected/1cc88905-e18f-4a2d-a503-4b4950cd27d3-kube-api-access-q42l2\") pod \"node-ca-74qh7\" (UID: \"1cc88905-e18f-4a2d-a503-4b4950cd27d3\") " pod="openshift-image-registry/node-ca-74qh7" Oct 30 23:05:40.430102 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429474 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-system-cni-dir\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.430102 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429488 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5bfb1046-f8b5-4035-bb35-173a1316ea0f-multus-daemon-config\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.430652 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429521 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-etc-kubernetes\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.430652 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429543 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-var-lib-cni-multus\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.430652 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429548 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lftl4\" (UniqueName: \"kubernetes.io/projected/5bfb1046-f8b5-4035-bb35-173a1316ea0f-kube-api-access-lftl4\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.430652 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429584 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t9v8\" (UniqueName: \"kubernetes.io/projected/706016da-3413-43e2-b06e-944572ca4106-kube-api-access-7t9v8\") pod \"iptables-alerter-khr89\" (UID: \"706016da-3413-43e2-b06e-944572ca4106\") " pod="openshift-network-operator/iptables-alerter-khr89" Oct 30 23:05:40.430652 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429622 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-var-lib-kubelet\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.430652 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429654 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-modprobe-d\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.430652 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429682 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-sysctl-d\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.430652 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429707 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-run\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.430652 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429733 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bfb1046-f8b5-4035-bb35-173a1316ea0f-cni-binary-copy\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.430652 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429759 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/706016da-3413-43e2-b06e-944572ca4106-iptables-alerter-script\") pod \"iptables-alerter-khr89\" (UID: \"706016da-3413-43e2-b06e-944572ca4106\") " pod="openshift-network-operator/iptables-alerter-khr89" Oct 30 23:05:40.430652 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429785 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cd7decf-5213-4360-a405-76d07c6ae3df-tmp-dir\") pod \"node-resolver-p84ng\" (UID: \"5cd7decf-5213-4360-a405-76d07c6ae3df\") " pod="openshift-dns/node-resolver-p84ng" Oct 30 23:05:40.430652 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429814 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-kubernetes\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.430652 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429861 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-etc-kubernetes\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.430652 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429870 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-kubernetes\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.430652 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429900 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-sysctl-conf\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.430652 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429430 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-run-k8s-cni-cncf-io\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.430652 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429924 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-var-lib-kubelet\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.430652 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.429987 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1cc88905-e18f-4a2d-a503-4b4950cd27d3-serviceca\") pod \"node-ca-74qh7\" (UID: \"1cc88905-e18f-4a2d-a503-4b4950cd27d3\") " pod="openshift-image-registry/node-ca-74qh7" Oct 30 23:05:40.431201 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.430005 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-modprobe-d\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.431201 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.430036 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-sysctl-d\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.431201 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.430037 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-sysctl-conf\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.431201 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.430101 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-run\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.431201 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.430229 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5bfb1046-f8b5-4035-bb35-173a1316ea0f-multus-daemon-config\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.431201 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.430309 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-sysconfig\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.431201 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.430350 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cd7decf-5213-4360-a405-76d07c6ae3df-tmp-dir\") pod \"node-resolver-p84ng\" (UID: \"5cd7decf-5213-4360-a405-76d07c6ae3df\") " pod="openshift-dns/node-resolver-p84ng" Oct 30 23:05:40.431201 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.430520 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bfb1046-f8b5-4035-bb35-173a1316ea0f-cni-binary-copy\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.431201 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.430615 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/706016da-3413-43e2-b06e-944572ca4106-iptables-alerter-script\") pod \"iptables-alerter-khr89\" (UID: \"706016da-3413-43e2-b06e-944572ca4106\") " pod="openshift-network-operator/iptables-alerter-khr89" Oct 30 23:05:40.431201 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.431149 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3838326a-857d-4511-8866-813015d85554-tmp\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.431501 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.431367 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3838326a-857d-4511-8866-813015d85554-etc-tuned\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.491591 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.491568 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t9v8\" (UniqueName: \"kubernetes.io/projected/706016da-3413-43e2-b06e-944572ca4106-kube-api-access-7t9v8\") pod \"iptables-alerter-khr89\" (UID: \"706016da-3413-43e2-b06e-944572ca4106\") " pod="openshift-network-operator/iptables-alerter-khr89" Oct 30 23:05:40.492026 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.492007 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tm5m\" (UniqueName: \"kubernetes.io/projected/5cd7decf-5213-4360-a405-76d07c6ae3df-kube-api-access-9tm5m\") pod \"node-resolver-p84ng\" (UID: \"5cd7decf-5213-4360-a405-76d07c6ae3df\") " pod="openshift-dns/node-resolver-p84ng" Oct 30 23:05:40.492138 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.492100 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjr2r\" (UniqueName: \"kubernetes.io/projected/3838326a-857d-4511-8866-813015d85554-kube-api-access-rjr2r\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.492312 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.492297 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lftl4\" (UniqueName: \"kubernetes.io/projected/5bfb1046-f8b5-4035-bb35-173a1316ea0f-kube-api-access-lftl4\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.492665 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.492647 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q42l2\" (UniqueName: \"kubernetes.io/projected/1cc88905-e18f-4a2d-a503-4b4950cd27d3-kube-api-access-q42l2\") pod \"node-ca-74qh7\" (UID: \"1cc88905-e18f-4a2d-a503-4b4950cd27d3\") " pod="openshift-image-registry/node-ca-74qh7" Oct 30 23:05:40.523837 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.523811 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9sp4c" Oct 30 23:05:40.534130 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:40.534097 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4d82728_b04b_4914_93d2_fce161acae36.slice/crio-a2e52979c7c424c66dfef4b2a3bb8bd8cb031109f65b823fb2725c331b66d839 WatchSource:0}: Error finding container a2e52979c7c424c66dfef4b2a3bb8bd8cb031109f65b823fb2725c331b66d839: Status 404 returned error can't find the container with id a2e52979c7c424c66dfef4b2a3bb8bd8cb031109f65b823fb2725c331b66d839 Oct 30 23:05:40.534763 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.534741 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:05:40.541937 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.541918 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:05:40.542038 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.541979 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:05:40.544216 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.544198 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zx7t9" Oct 30 23:05:40.544832 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:40.544814 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2861d402_9b72_4907_b8c1_7199ee3a8e92.slice/crio-c9afe92d8ad57aa5fdeba3b2c942001aeb12544434f72c6727b33f2f0802d26d WatchSource:0}: Error finding container c9afe92d8ad57aa5fdeba3b2c942001aeb12544434f72c6727b33f2f0802d26d: Status 404 returned error can't find the container with id c9afe92d8ad57aa5fdeba3b2c942001aeb12544434f72c6727b33f2f0802d26d Oct 30 23:05:40.554698 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.554676 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p84ng" Oct 30 23:05:40.557630 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.557613 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-74qh7" Oct 30 23:05:40.559755 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:40.559734 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3838326a_857d_4511_8866_813015d85554.slice/crio-e40b78f6fd35a5f24bbc1d6c322d401a5a25c77284976c1abcd4752f2c0e58a6 WatchSource:0}: Error finding container e40b78f6fd35a5f24bbc1d6c322d401a5a25c77284976c1abcd4752f2c0e58a6: Status 404 returned error can't find the container with id e40b78f6fd35a5f24bbc1d6c322d401a5a25c77284976c1abcd4752f2c0e58a6 Oct 30 23:05:40.562404 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.562385 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:05:40.564920 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.564901 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-khr89" Oct 30 23:05:40.568359 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:40.568333 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20eb8eea_5fd8_42ab_bd78_8abc77f84eb2.slice/crio-4dd83f85253bebe12c121175d879cd3fd7e9e87a39ffe8f058ac37eee18f8c9c WatchSource:0}: Error finding container 4dd83f85253bebe12c121175d879cd3fd7e9e87a39ffe8f058ac37eee18f8c9c: Status 404 returned error can't find the container with id 4dd83f85253bebe12c121175d879cd3fd7e9e87a39ffe8f058ac37eee18f8c9c Oct 30 23:05:40.569273 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:40.569235 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bfb1046_f8b5_4035_bb35_173a1316ea0f.slice/crio-e5deb16a572b4a313257ac5667bbc9a6f23ad2e38dfc6313dbf21bf25c868847 WatchSource:0}: Error finding container e5deb16a572b4a313257ac5667bbc9a6f23ad2e38dfc6313dbf21bf25c868847: Status 404 returned error can't find the container with id e5deb16a572b4a313257ac5667bbc9a6f23ad2e38dfc6313dbf21bf25c868847 Oct 30 23:05:40.572086 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:05:40.570471 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd7decf_5213_4360_a405_76d07c6ae3df.slice/crio-e50f5624186bdf7c4bb7e28866f82676e6888120749c7b1056e4e72297fb3b6a WatchSource:0}: Error finding container e50f5624186bdf7c4bb7e28866f82676e6888120749c7b1056e4e72297fb3b6a: Status 404 returned error can't find the container with id e50f5624186bdf7c4bb7e28866f82676e6888120749c7b1056e4e72297fb3b6a Oct 30 23:05:40.833543 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.833456 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/322e3a9b-911f-4cd8-8f43-71d56152e917-original-pull-secret\") pod \"global-pull-secret-syncer-7vrpr\" (UID: \"322e3a9b-911f-4cd8-8f43-71d56152e917\") " pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:05:40.833543 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.833527 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs\") pod \"network-metrics-daemon-htqtp\" (UID: \"dec2c5fa-4b9a-47ed-b005-04874761f419\") " pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:05:40.833774 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:40.833668 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:05:40.833774 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:40.833726 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs podName:dec2c5fa-4b9a-47ed-b005-04874761f419 nodeName:}" failed. No retries permitted until 2026-10-30 23:05:41.833707406 +0000 UTC m=+4.069132192 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs") pod "network-metrics-daemon-htqtp" (UID: "dec2c5fa-4b9a-47ed-b005-04874761f419") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:05:40.833900 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:40.833825 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:05:40.833900 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:40.833882 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/322e3a9b-911f-4cd8-8f43-71d56152e917-original-pull-secret podName:322e3a9b-911f-4cd8-8f43-71d56152e917 nodeName:}" failed. No retries permitted until 2026-10-30 23:05:41.83386992 +0000 UTC m=+4.069294705 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/322e3a9b-911f-4cd8-8f43-71d56152e917-original-pull-secret") pod "global-pull-secret-syncer-7vrpr" (UID: "322e3a9b-911f-4cd8-8f43-71d56152e917") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:05:40.934768 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:40.934728 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9nfl\" (UniqueName: \"kubernetes.io/projected/4f7d22b5-256e-47d8-947f-b89b47f14d08-kube-api-access-l9nfl\") pod \"network-check-target-xdlsv\" (UID: \"4f7d22b5-256e-47d8-947f-b89b47f14d08\") " pod="openshift-network-diagnostics/network-check-target-xdlsv" Oct 30 23:05:40.934957 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:40.934926 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:05:40.934957 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:40.934945 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:05:40.934957 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:40.934958 2570 projected.go:196] Error preparing data for projected volume kube-api-access-l9nfl for pod openshift-network-diagnostics/network-check-target-xdlsv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:05:40.935161 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:40.935015 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f7d22b5-256e-47d8-947f-b89b47f14d08-kube-api-access-l9nfl podName:4f7d22b5-256e-47d8-947f-b89b47f14d08 nodeName:}" failed. No retries permitted until 2026-10-30 23:05:41.934995687 +0000 UTC m=+4.170420472 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-l9nfl" (UniqueName: "kubernetes.io/projected/4f7d22b5-256e-47d8-947f-b89b47f14d08-kube-api-access-l9nfl") pod "network-check-target-xdlsv" (UID: "4f7d22b5-256e-47d8-947f-b89b47f14d08") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:05:41.193250 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:41.193160 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9cltq" event={"ID":"c4368706-761a-4f6e-9dda-a8087a5f4fdc","Type":"ContainerStarted","Data":"ce66ace81ace169fbc0c516afacbdf9d1e4f43fb8bff281b5ae5dcfacf220b1d"} Oct 30 23:05:41.194506 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:41.194475 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-74qh7" event={"ID":"1cc88905-e18f-4a2d-a503-4b4950cd27d3","Type":"ContainerStarted","Data":"93ca6840cfd406c206a6093b26ff3ae4d0b6daef2ba2a11b345def032d54cd6f"} Oct 30 23:05:41.195885 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:41.195857 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p84ng" event={"ID":"5cd7decf-5213-4360-a405-76d07c6ae3df","Type":"ContainerStarted","Data":"e50f5624186bdf7c4bb7e28866f82676e6888120749c7b1056e4e72297fb3b6a"} Oct 30 23:05:41.197306 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:41.197271 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" event={"ID":"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2","Type":"ContainerStarted","Data":"4dd83f85253bebe12c121175d879cd3fd7e9e87a39ffe8f058ac37eee18f8c9c"} Oct 30 23:05:41.198765 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:41.198738 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9sp4c" event={"ID":"e4d82728-b04b-4914-93d2-fce161acae36","Type":"ContainerStarted","Data":"a2e52979c7c424c66dfef4b2a3bb8bd8cb031109f65b823fb2725c331b66d839"} Oct 30 23:05:41.199972 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:41.199951 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-khr89" event={"ID":"706016da-3413-43e2-b06e-944572ca4106","Type":"ContainerStarted","Data":"df5f40c01fa381a557e544f6888feca5bb2fe273560e70b3fa5669ae9a73ab02"} Oct 30 23:05:41.201216 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:41.201193 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zx7t9" event={"ID":"5bfb1046-f8b5-4035-bb35-173a1316ea0f","Type":"ContainerStarted","Data":"e5deb16a572b4a313257ac5667bbc9a6f23ad2e38dfc6313dbf21bf25c868847"} Oct 30 23:05:41.202536 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:41.202515 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z4l28" event={"ID":"3838326a-857d-4511-8866-813015d85554","Type":"ContainerStarted","Data":"e40b78f6fd35a5f24bbc1d6c322d401a5a25c77284976c1abcd4752f2c0e58a6"} Oct 30 23:05:41.204214 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:41.204188 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" event={"ID":"2861d402-9b72-4907-b8c1-7199ee3a8e92","Type":"ContainerStarted","Data":"c9afe92d8ad57aa5fdeba3b2c942001aeb12544434f72c6727b33f2f0802d26d"} Oct 30 23:05:41.307347 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:41.307251 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:00:40 +0000 UTC" deadline="2028-04-22 07:12:59.095787265 +0000 UTC" Oct 30 23:05:41.307347 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:41.307315 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12944h7m17.788477886s" Oct 30 23:05:41.843860 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:41.843462 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/322e3a9b-911f-4cd8-8f43-71d56152e917-original-pull-secret\") pod \"global-pull-secret-syncer-7vrpr\" (UID: \"322e3a9b-911f-4cd8-8f43-71d56152e917\") " pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:05:41.843860 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:41.843534 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs\") pod \"network-metrics-daemon-htqtp\" (UID: \"dec2c5fa-4b9a-47ed-b005-04874761f419\") " pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:05:41.843860 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:41.843671 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:05:41.843860 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:41.843728 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs podName:dec2c5fa-4b9a-47ed-b005-04874761f419 nodeName:}" failed. No retries permitted until 2026-10-30 23:05:43.843709304 +0000 UTC m=+6.079134093 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs") pod "network-metrics-daemon-htqtp" (UID: "dec2c5fa-4b9a-47ed-b005-04874761f419") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:05:41.843860 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:41.843794 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:05:41.843860 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:41.843827 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/322e3a9b-911f-4cd8-8f43-71d56152e917-original-pull-secret podName:322e3a9b-911f-4cd8-8f43-71d56152e917 nodeName:}" failed. No retries permitted until 2026-10-30 23:05:43.843816214 +0000 UTC m=+6.079241001 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/322e3a9b-911f-4cd8-8f43-71d56152e917-original-pull-secret") pod "global-pull-secret-syncer-7vrpr" (UID: "322e3a9b-911f-4cd8-8f43-71d56152e917") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:05:41.943943 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:41.943904 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9nfl\" (UniqueName: \"kubernetes.io/projected/4f7d22b5-256e-47d8-947f-b89b47f14d08-kube-api-access-l9nfl\") pod \"network-check-target-xdlsv\" (UID: \"4f7d22b5-256e-47d8-947f-b89b47f14d08\") " pod="openshift-network-diagnostics/network-check-target-xdlsv" Oct 30 23:05:41.944148 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:41.944108 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:05:41.944148 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:41.944135 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:05:41.944148 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:41.944149 2570 projected.go:196] Error preparing data for projected volume kube-api-access-l9nfl for pod openshift-network-diagnostics/network-check-target-xdlsv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:05:41.944327 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:41.944207 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f7d22b5-256e-47d8-947f-b89b47f14d08-kube-api-access-l9nfl podName:4f7d22b5-256e-47d8-947f-b89b47f14d08 nodeName:}" failed. No retries permitted until 2026-10-30 23:05:43.944188821 +0000 UTC m=+6.179613590 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-l9nfl" (UniqueName: "kubernetes.io/projected/4f7d22b5-256e-47d8-947f-b89b47f14d08-kube-api-access-l9nfl") pod "network-check-target-xdlsv" (UID: "4f7d22b5-256e-47d8-947f-b89b47f14d08") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:05:42.188344 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:42.188274 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:05:42.188495 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:42.188429 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-htqtp" podUID="dec2c5fa-4b9a-47ed-b005-04874761f419" Oct 30 23:05:42.188635 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:42.188523 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdlsv" Oct 30 23:05:42.188635 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:42.188606 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xdlsv" podUID="4f7d22b5-256e-47d8-947f-b89b47f14d08" Oct 30 23:05:42.189171 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:42.189139 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:05:42.189288 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:42.189244 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vrpr" podUID="322e3a9b-911f-4cd8-8f43-71d56152e917" Oct 30 23:05:42.307965 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:42.307881 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:00:40 +0000 UTC" deadline="2028-08-04 09:51:56.96836708 +0000 UTC" Oct 30 23:05:42.307965 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:42.307961 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15442h46m14.660410702s" Oct 30 23:05:43.222541 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:43.222252 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-177.us-west-1.compute.internal" event={"ID":"f343176fefb75130c5aad85365de861f","Type":"ContainerStarted","Data":"1dc97c46d12b832a1d1b2fb6e51032133d7255127ee94f24663d140ac07501e7"} Oct 30 23:05:43.862362 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:43.862134 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/322e3a9b-911f-4cd8-8f43-71d56152e917-original-pull-secret\") pod \"global-pull-secret-syncer-7vrpr\" (UID: \"322e3a9b-911f-4cd8-8f43-71d56152e917\") " pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:05:43.862362 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:43.862198 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs\") pod \"network-metrics-daemon-htqtp\" (UID: \"dec2c5fa-4b9a-47ed-b005-04874761f419\") " pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:05:43.862362 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:43.862271 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:05:43.862362 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:43.862335 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/322e3a9b-911f-4cd8-8f43-71d56152e917-original-pull-secret podName:322e3a9b-911f-4cd8-8f43-71d56152e917 nodeName:}" failed. No retries permitted until 2026-10-30 23:05:47.862313282 +0000 UTC m=+10.097738052 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/322e3a9b-911f-4cd8-8f43-71d56152e917-original-pull-secret") pod "global-pull-secret-syncer-7vrpr" (UID: "322e3a9b-911f-4cd8-8f43-71d56152e917") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:05:43.862362 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:43.862338 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:05:43.863269 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:43.862374 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs podName:dec2c5fa-4b9a-47ed-b005-04874761f419 nodeName:}" failed. No retries permitted until 2026-10-30 23:05:47.86236282 +0000 UTC m=+10.097787605 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs") pod "network-metrics-daemon-htqtp" (UID: "dec2c5fa-4b9a-47ed-b005-04874761f419") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:05:43.964145 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:43.963425 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9nfl\" (UniqueName: \"kubernetes.io/projected/4f7d22b5-256e-47d8-947f-b89b47f14d08-kube-api-access-l9nfl\") pod \"network-check-target-xdlsv\" (UID: \"4f7d22b5-256e-47d8-947f-b89b47f14d08\") " pod="openshift-network-diagnostics/network-check-target-xdlsv" Oct 30 23:05:43.964145 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:43.963610 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:05:43.964145 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:43.963634 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:05:43.964145 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:43.963648 2570 projected.go:196] Error preparing data for projected volume kube-api-access-l9nfl for pod openshift-network-diagnostics/network-check-target-xdlsv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:05:43.964145 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:43.963724 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f7d22b5-256e-47d8-947f-b89b47f14d08-kube-api-access-l9nfl podName:4f7d22b5-256e-47d8-947f-b89b47f14d08 nodeName:}" failed. No retries permitted until 2026-10-30 23:05:47.963704228 +0000 UTC m=+10.199129014 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-l9nfl" (UniqueName: "kubernetes.io/projected/4f7d22b5-256e-47d8-947f-b89b47f14d08-kube-api-access-l9nfl") pod "network-check-target-xdlsv" (UID: "4f7d22b5-256e-47d8-947f-b89b47f14d08") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:05:44.184824 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:44.184771 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:05:44.184824 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:44.184792 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:05:44.185227 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:44.184777 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdlsv" Oct 30 23:05:44.185227 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:44.185158 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-htqtp" podUID="dec2c5fa-4b9a-47ed-b005-04874761f419" Oct 30 23:05:44.185333 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:44.185316 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vrpr" podUID="322e3a9b-911f-4cd8-8f43-71d56152e917" Oct 30 23:05:44.185496 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:44.185433 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xdlsv" podUID="4f7d22b5-256e-47d8-947f-b89b47f14d08" Oct 30 23:05:44.237873 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:44.236797 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9cltq" event={"ID":"c4368706-761a-4f6e-9dda-a8087a5f4fdc","Type":"ContainerStarted","Data":"6130681a4911f769eca5e7dd4bf58582e51e3c147984c1f714d5de9a4cdaa200"} Oct 30 23:05:44.245501 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:44.244396 2570 generic.go:356] "Generic (PLEG): container finished" podID="5d800e6e610855f53a5c54432ef51ba0" containerID="b1c73c5de5730069ed0ac6e21df2f41be2cd1dd3cc6448bd221f32fc113f1817" exitCode=0 Oct 30 23:05:44.245501 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:44.245403 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal" event={"ID":"5d800e6e610855f53a5c54432ef51ba0","Type":"ContainerDied","Data":"b1c73c5de5730069ed0ac6e21df2f41be2cd1dd3cc6448bd221f32fc113f1817"} Oct 30 23:05:44.367334 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:44.367218 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-177.us-west-1.compute.internal" podStartSLOduration=4.367198712 podStartE2EDuration="4.367198712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:05:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:05:43.361169786 +0000 UTC m=+5.596594579" watchObservedRunningTime="2026-10-30 23:05:44.367198712 +0000 UTC m=+6.602623508" Oct 30 23:05:45.252232 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:45.252174 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal" event={"ID":"5d800e6e610855f53a5c54432ef51ba0","Type":"ContainerStarted","Data":"06190e290792cc7dbc70297ed15f7f3f7656a69ade8ba2413405f9f1435da458"} Oct 30 23:05:45.259851 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:45.259818 2570 generic.go:356] "Generic (PLEG): container finished" podID="c4368706-761a-4f6e-9dda-a8087a5f4fdc" containerID="6130681a4911f769eca5e7dd4bf58582e51e3c147984c1f714d5de9a4cdaa200" exitCode=0 Oct 30 23:05:45.260040 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:45.259871 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9cltq" event={"ID":"c4368706-761a-4f6e-9dda-a8087a5f4fdc","Type":"ContainerDied","Data":"6130681a4911f769eca5e7dd4bf58582e51e3c147984c1f714d5de9a4cdaa200"} Oct 30 23:05:45.377578 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:45.377515 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal" podStartSLOduration=5.377498986 podStartE2EDuration="5.377498986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:05:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:05:45.376985962 +0000 UTC m=+7.612410768" watchObservedRunningTime="2026-10-30 23:05:45.377498986 +0000 UTC m=+7.612923779" Oct 30 23:05:46.188482 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:46.187887 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:05:46.188482 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:46.188030 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-htqtp" podUID="dec2c5fa-4b9a-47ed-b005-04874761f419" Oct 30 23:05:46.188482 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:46.188442 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdlsv" Oct 30 23:05:46.188710 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:46.188536 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xdlsv" podUID="4f7d22b5-256e-47d8-947f-b89b47f14d08" Oct 30 23:05:46.188710 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:46.188592 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:05:46.188710 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:46.188665 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vrpr" podUID="322e3a9b-911f-4cd8-8f43-71d56152e917" Oct 30 23:05:46.271125 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:46.271084 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9sp4c" event={"ID":"e4d82728-b04b-4914-93d2-fce161acae36","Type":"ContainerStarted","Data":"9f8437bc39ca0b0a469e3f3c5579f9a12ce8b77e7ae7c2a9ecc11cc82d5b7e80"} Oct 30 23:05:46.393951 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:46.393892 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9sp4c" podStartSLOduration=2.456519566 podStartE2EDuration="7.393869969s" totalImagesPullingTime="4.937350403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:05:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:05:46.393576481 +0000 UTC m=+8.629001277" watchObservedRunningTime="2026-10-30 23:05:46.393869969 +0000 UTC m=+8.629294763" Oct 30 23:05:47.280291 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:47.280167 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-74qh7" event={"ID":"1cc88905-e18f-4a2d-a503-4b4950cd27d3","Type":"ContainerStarted","Data":"6c59d64c4f3e9fe895882f66908902cb7318c5b5c096e607eafa40417325bd5d"} Oct 30 23:05:47.652342 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:47.651959 2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9sp4c" Oct 30 23:05:47.652590 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:47.652567 2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9sp4c" Oct 30 23:05:47.777774 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:47.777666 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-74qh7" podStartSLOduration=2.3660376149999998 podStartE2EDuration="8.777642117s" totalImagesPullingTime="6.411604502s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:05:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:05:47.40387452 +0000 UTC m=+9.639299316" watchObservedRunningTime="2026-10-30 23:05:47.777642117 +0000 UTC m=+10.013066912" Oct 30 23:05:47.909875 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:47.909839 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/322e3a9b-911f-4cd8-8f43-71d56152e917-original-pull-secret\") pod \"global-pull-secret-syncer-7vrpr\" (UID: \"322e3a9b-911f-4cd8-8f43-71d56152e917\") " pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:05:47.910051 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:47.909909 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs\") pod \"network-metrics-daemon-htqtp\" (UID: \"dec2c5fa-4b9a-47ed-b005-04874761f419\") " pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:05:47.910051 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:47.910027 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:05:47.910202 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:47.910095 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs podName:dec2c5fa-4b9a-47ed-b005-04874761f419 nodeName:}" failed. No retries permitted until 2026-10-30 23:05:55.910080413 +0000 UTC m=+18.145505213 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs") pod "network-metrics-daemon-htqtp" (UID: "dec2c5fa-4b9a-47ed-b005-04874761f419") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:05:47.910321 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:47.910302 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:05:47.910382 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:47.910358 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/322e3a9b-911f-4cd8-8f43-71d56152e917-original-pull-secret podName:322e3a9b-911f-4cd8-8f43-71d56152e917 nodeName:}" failed. No retries permitted until 2026-10-30 23:05:55.910341331 +0000 UTC m=+18.145766119 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/322e3a9b-911f-4cd8-8f43-71d56152e917-original-pull-secret") pod "global-pull-secret-syncer-7vrpr" (UID: "322e3a9b-911f-4cd8-8f43-71d56152e917") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:05:48.011235 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:48.011193 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9nfl\" (UniqueName: \"kubernetes.io/projected/4f7d22b5-256e-47d8-947f-b89b47f14d08-kube-api-access-l9nfl\") pod \"network-check-target-xdlsv\" (UID: \"4f7d22b5-256e-47d8-947f-b89b47f14d08\") " pod="openshift-network-diagnostics/network-check-target-xdlsv" Oct 30 23:05:48.011422 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:48.011398 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:05:48.011422 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:48.011422 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:05:48.011543 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:48.011438 2570 projected.go:196] Error preparing data for projected volume kube-api-access-l9nfl for pod openshift-network-diagnostics/network-check-target-xdlsv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:05:48.011543 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:48.011501 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f7d22b5-256e-47d8-947f-b89b47f14d08-kube-api-access-l9nfl podName:4f7d22b5-256e-47d8-947f-b89b47f14d08 nodeName:}" failed. No retries permitted until 2026-10-30 23:05:56.011482461 +0000 UTC m=+18.246907231 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-l9nfl" (UniqueName: "kubernetes.io/projected/4f7d22b5-256e-47d8-947f-b89b47f14d08-kube-api-access-l9nfl") pod "network-check-target-xdlsv" (UID: "4f7d22b5-256e-47d8-947f-b89b47f14d08") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:05:48.187180 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:48.187075 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:05:48.187343 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:48.187253 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-htqtp" podUID="dec2c5fa-4b9a-47ed-b005-04874761f419" Oct 30 23:05:48.187682 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:48.187658 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdlsv" Oct 30 23:05:48.187813 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:48.187762 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xdlsv" podUID="4f7d22b5-256e-47d8-947f-b89b47f14d08" Oct 30 23:05:48.187878 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:48.187833 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:05:48.187934 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:48.187914 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vrpr" podUID="322e3a9b-911f-4cd8-8f43-71d56152e917" Oct 30 23:05:48.286046 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:48.286010 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" event={"ID":"2861d402-9b72-4907-b8c1-7199ee3a8e92","Type":"ContainerStarted","Data":"31d704624c1ec29422866c3c6fd6b423835d48a6421463272d5969879ad65f89"} Oct 30 23:05:49.300621 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:49.300552 2570 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 23:05:49.301662 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:49.301211 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zx7t9" event={"ID":"5bfb1046-f8b5-4035-bb35-173a1316ea0f","Type":"ContainerStarted","Data":"53c645de5e96392c65bae993590288c63232e613768d11e67242783c1d345b27"} Oct 30 23:05:49.565896 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:49.565871 2570 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 23:05:50.187151 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:50.186190 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdlsv" Oct 30 23:05:50.187151 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:50.186542 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xdlsv" podUID="4f7d22b5-256e-47d8-947f-b89b47f14d08" Oct 30 23:05:50.187151 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:50.186277 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:05:50.187151 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:50.186658 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-htqtp" podUID="dec2c5fa-4b9a-47ed-b005-04874761f419" Oct 30 23:05:50.187151 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:50.186268 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:05:50.187151 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:50.186742 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vrpr" podUID="322e3a9b-911f-4cd8-8f43-71d56152e917" Oct 30 23:05:50.192257 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:50.192156 2570 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T23:05:49.56589421Z","UUID":"8258a55a-e57f-4900-bdb2-268f72a97a8f","Handler":null,"Name":"","Endpoint":""} Oct 30 23:05:50.193643 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:50.193617 2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 23:05:50.193751 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:50.193650 2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 23:05:50.305234 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:50.305199 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" event={"ID":"2861d402-9b72-4907-b8c1-7199ee3a8e92","Type":"ContainerStarted","Data":"eaa05632080110cfc006e7fa1acc2130dc3d68d17e677ed97ba27e2822a65947"} Oct 30 23:05:50.308322 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:50.308093 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p84ng" event={"ID":"5cd7decf-5213-4360-a405-76d07c6ae3df","Type":"ContainerStarted","Data":"e71e946a7ff08130dae834ae279ff3da9afd4200b94866861a5ab32d9dd0092f"} Oct 30 23:05:50.424901 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:50.424848 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-zx7t9" podStartSLOduration=2.968961994 podStartE2EDuration="11.42483066s" totalImagesPullingTime="8.455868666s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:05:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:05:49.430414048 +0000 UTC m=+11.665838843" watchObservedRunningTime="2026-10-30 23:05:50.42483066 +0000 UTC m=+12.660255461" Oct 30 23:05:50.425640 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:50.425604 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-p84ng" podStartSLOduration=2.15036419 podStartE2EDuration="11.425592734s" totalImagesPullingTime="9.275228544s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:05:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:05:50.42455579 +0000 UTC m=+12.659980579" watchObservedRunningTime="2026-10-30 23:05:50.425592734 +0000 UTC m=+12.661017524" Oct 30 23:05:51.311548 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:51.311513 2570 generic.go:356] "Generic (PLEG): container finished" podID="c4368706-761a-4f6e-9dda-a8087a5f4fdc" containerID="1f0300bb95e03d0a743ac258313f58bf64fe4f61739d233c8856bffbe41f47ad" exitCode=0 Oct 30 23:05:51.312239 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:51.311603 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9cltq" event={"ID":"c4368706-761a-4f6e-9dda-a8087a5f4fdc","Type":"ContainerDied","Data":"1f0300bb95e03d0a743ac258313f58bf64fe4f61739d233c8856bffbe41f47ad"} Oct 30 23:05:52.185357 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:52.185330 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdlsv" Oct 30 23:05:52.185506 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:52.185398 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:05:52.185506 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:52.185398 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:05:52.185637 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:52.185497 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vrpr" podUID="322e3a9b-911f-4cd8-8f43-71d56152e917" Oct 30 23:05:52.185637 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:52.185610 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xdlsv" podUID="4f7d22b5-256e-47d8-947f-b89b47f14d08" Oct 30 23:05:52.185736 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:52.185702 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-htqtp" podUID="dec2c5fa-4b9a-47ed-b005-04874761f419" Oct 30 23:05:52.232793 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:52.232761 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9sp4c" Oct 30 23:05:52.232942 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:52.232873 2570 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 23:05:52.233557 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:52.233533 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9sp4c" Oct 30 23:05:52.315739 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:52.315673 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-khr89" event={"ID":"706016da-3413-43e2-b06e-944572ca4106","Type":"ContainerStarted","Data":"1b8e139c43866b16bb7dd92d524f3ea591d56023366c222cee260a62ea403693"} Oct 30 23:05:52.317419 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:52.317390 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z4l28" event={"ID":"3838326a-857d-4511-8866-813015d85554","Type":"ContainerStarted","Data":"13490e40f9b69d305b94f0ac889d33bdcc4c36dd7f1d6381422b45fc8f808879"} Oct 30 23:05:52.319948 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:52.319915 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" event={"ID":"2861d402-9b72-4907-b8c1-7199ee3a8e92","Type":"ContainerStarted","Data":"acde7d5354e0ebe65c3f4018ab7cc56669aa9c7a917ae34ba44aca951bcbfc6f"} Oct 30 23:05:52.461321 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:52.461270 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-khr89" podStartSLOduration=4.116259371 podStartE2EDuration="13.461254454s" totalImagesPullingTime="9.344995083s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:05:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:05:52.460897373 +0000 UTC m=+14.696322163" watchObservedRunningTime="2026-10-30 23:05:52.461254454 +0000 UTC m=+14.696679246" Oct 30 23:05:52.696162 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:52.696112 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" podStartSLOduration=3.298625521 podStartE2EDuration="13.696095941s" totalImagesPullingTime="10.39747042s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:05:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:05:52.577614822 +0000 UTC m=+14.813039615" watchObservedRunningTime="2026-10-30 23:05:52.696095941 +0000 UTC m=+14.931520731" Oct 30 23:05:52.696396 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:52.696369 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-z4l28" podStartSLOduration=2.200982792 podStartE2EDuration="13.696360242s" totalImagesPullingTime="11.49537745s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:05:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:05:52.695909908 +0000 UTC m=+14.931334701" watchObservedRunningTime="2026-10-30 23:05:52.696360242 +0000 UTC m=+14.931785036" Oct 30 23:05:53.324875 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:53.324840 2570 generic.go:356] "Generic (PLEG): container finished" podID="c4368706-761a-4f6e-9dda-a8087a5f4fdc" containerID="b495f09c9e4750943f4b3f475cb0a79ede3e9a2ed508965ba0958c5fbb8a6c4e" exitCode=0 Oct 30 23:05:53.325388 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:53.324939 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9cltq" event={"ID":"c4368706-761a-4f6e-9dda-a8087a5f4fdc","Type":"ContainerDied","Data":"b495f09c9e4750943f4b3f475cb0a79ede3e9a2ed508965ba0958c5fbb8a6c4e"} Oct 30 23:05:54.185391 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:54.185156 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdlsv" Oct 30 23:05:54.185581 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:54.185165 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:05:54.185581 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:54.185490 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xdlsv" podUID="4f7d22b5-256e-47d8-947f-b89b47f14d08" Oct 30 23:05:54.185581 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:54.185571 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-htqtp" podUID="dec2c5fa-4b9a-47ed-b005-04874761f419" Oct 30 23:05:54.185745 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:54.185215 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:05:54.185745 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:54.185663 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vrpr" podUID="322e3a9b-911f-4cd8-8f43-71d56152e917" Oct 30 23:05:55.333380 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:55.333346 2570 generic.go:356] "Generic (PLEG): container finished" podID="c4368706-761a-4f6e-9dda-a8087a5f4fdc" containerID="77f2c64a49759466d37dfd95ca5ffe95365a543bb7af3853b857e74932b92e03" exitCode=0 Oct 30 23:05:55.333838 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:55.333399 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9cltq" event={"ID":"c4368706-761a-4f6e-9dda-a8087a5f4fdc","Type":"ContainerDied","Data":"77f2c64a49759466d37dfd95ca5ffe95365a543bb7af3853b857e74932b92e03"} Oct 30 23:05:55.976792 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:55.976766 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/322e3a9b-911f-4cd8-8f43-71d56152e917-original-pull-secret\") pod \"global-pull-secret-syncer-7vrpr\" (UID: \"322e3a9b-911f-4cd8-8f43-71d56152e917\") " pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:05:55.977081 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:55.976807 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs\") pod \"network-metrics-daemon-htqtp\" (UID: \"dec2c5fa-4b9a-47ed-b005-04874761f419\") " pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:05:55.977081 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:55.976903 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:05:55.977081 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:55.976920 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:05:55.977081 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:55.976952 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs podName:dec2c5fa-4b9a-47ed-b005-04874761f419 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:11.97693613 +0000 UTC m=+34.212360899 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs") pod "network-metrics-daemon-htqtp" (UID: "dec2c5fa-4b9a-47ed-b005-04874761f419") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:05:55.977081 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:55.976976 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/322e3a9b-911f-4cd8-8f43-71d56152e917-original-pull-secret podName:322e3a9b-911f-4cd8-8f43-71d56152e917 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:11.976959137 +0000 UTC m=+34.212383923 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/322e3a9b-911f-4cd8-8f43-71d56152e917-original-pull-secret") pod "global-pull-secret-syncer-7vrpr" (UID: "322e3a9b-911f-4cd8-8f43-71d56152e917") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:05:56.077810 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:56.077787 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9nfl\" (UniqueName: \"kubernetes.io/projected/4f7d22b5-256e-47d8-947f-b89b47f14d08-kube-api-access-l9nfl\") pod \"network-check-target-xdlsv\" (UID: \"4f7d22b5-256e-47d8-947f-b89b47f14d08\") " pod="openshift-network-diagnostics/network-check-target-xdlsv" Oct 30 23:05:56.077928 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:56.077914 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:05:56.077968 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:56.077934 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:05:56.077968 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:56.077944 2570 projected.go:196] Error preparing data for projected volume kube-api-access-l9nfl for pod openshift-network-diagnostics/network-check-target-xdlsv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:05:56.078022 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:56.077985 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f7d22b5-256e-47d8-947f-b89b47f14d08-kube-api-access-l9nfl podName:4f7d22b5-256e-47d8-947f-b89b47f14d08 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:12.077973942 +0000 UTC m=+34.313398724 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-l9nfl" (UniqueName: "kubernetes.io/projected/4f7d22b5-256e-47d8-947f-b89b47f14d08-kube-api-access-l9nfl") pod "network-check-target-xdlsv" (UID: "4f7d22b5-256e-47d8-947f-b89b47f14d08") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:05:56.185024 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:56.185003 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdlsv" Oct 30 23:05:56.185024 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:56.185011 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:05:56.185207 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:56.185009 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:05:56.185207 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:56.185175 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vrpr" podUID="322e3a9b-911f-4cd8-8f43-71d56152e917" Oct 30 23:05:56.185281 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:56.185244 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-htqtp" podUID="dec2c5fa-4b9a-47ed-b005-04874761f419" Oct 30 23:05:56.185344 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:56.185327 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xdlsv" podUID="4f7d22b5-256e-47d8-947f-b89b47f14d08" Oct 30 23:05:56.338169 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:56.337973 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 30 23:05:56.338534 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:56.338503 2570 generic.go:356] "Generic (PLEG): container finished" podID="20eb8eea-5fd8-42ab-bd78-8abc77f84eb2" containerID="399eb8749765d7f02db13a1f1488b6b722b03162d5f0119e102148b1ad22f3b7" exitCode=1 Oct 30 23:05:56.338590 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:56.338549 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" event={"ID":"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2","Type":"ContainerStarted","Data":"4008104fc4a8a3a5536070f151eee777c3c04fbc95ace6b3c8631517dfaccfac"} Oct 30 23:05:56.338707 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:56.338631 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" event={"ID":"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2","Type":"ContainerStarted","Data":"62e3da700e3202a6df9aedc848b606139612ed146f6cab9065141339f89af99c"} Oct 30 23:05:56.338707 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:56.338661 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" event={"ID":"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2","Type":"ContainerStarted","Data":"a1fb9a27accaffcd3f4908a6f0963096fa499674246c2b60b7b2040bf5d98f31"} Oct 30 23:05:56.338707 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:56.338689 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" event={"ID":"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2","Type":"ContainerDied","Data":"399eb8749765d7f02db13a1f1488b6b722b03162d5f0119e102148b1ad22f3b7"} Oct 30 23:05:56.338842 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:56.338715 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" event={"ID":"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2","Type":"ContainerStarted","Data":"95b328534f8e89040ae24bbd1ecf39e63a44e6efb08a4b55dee92e1bb5b36720"} Oct 30 23:05:57.346261 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:57.346241 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 30 23:05:57.346694 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:57.346600 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" event={"ID":"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2","Type":"ContainerStarted","Data":"7ce0e8d66ecd6b39bf378a2b817aeb530b46d49058ae994ba96267af521413af"} Oct 30 23:05:58.185297 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:58.185269 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:05:58.185477 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:58.185377 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdlsv" Oct 30 23:05:58.185477 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:58.185426 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:05:58.185477 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:58.185448 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-htqtp" podUID="dec2c5fa-4b9a-47ed-b005-04874761f419" Oct 30 23:05:58.185669 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:58.185495 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vrpr" podUID="322e3a9b-911f-4cd8-8f43-71d56152e917" Oct 30 23:05:58.185669 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:05:58.185586 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xdlsv" podUID="4f7d22b5-256e-47d8-947f-b89b47f14d08" Oct 30 23:05:58.350709 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:58.350682 2570 generic.go:356] "Generic (PLEG): container finished" podID="c4368706-761a-4f6e-9dda-a8087a5f4fdc" containerID="be7f65ba4c1b84761a272cafb72344846bdc4c02c9e3931010e96d3e18a62792" exitCode=0 Oct 30 23:05:58.351209 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:58.350723 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9cltq" event={"ID":"c4368706-761a-4f6e-9dda-a8087a5f4fdc","Type":"ContainerDied","Data":"be7f65ba4c1b84761a272cafb72344846bdc4c02c9e3931010e96d3e18a62792"} Oct 30 23:05:59.355950 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:59.355912 2570 generic.go:356] "Generic (PLEG): container finished" podID="c4368706-761a-4f6e-9dda-a8087a5f4fdc" containerID="0e7f7a052841a0951ab3e18e295072dcb8e2f722fe318e4b586cdc60290545d6" exitCode=0 Oct 30 23:05:59.356453 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:59.355998 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9cltq" event={"ID":"c4368706-761a-4f6e-9dda-a8087a5f4fdc","Type":"ContainerDied","Data":"0e7f7a052841a0951ab3e18e295072dcb8e2f722fe318e4b586cdc60290545d6"} Oct 30 23:05:59.359067 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:59.358971 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 30 23:05:59.359365 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:05:59.359343 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" event={"ID":"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2","Type":"ContainerStarted","Data":"b48f2aebf36f7aaccbfb3919f76b537589937d713ba445d2401b16ba6bfcc8c7"} Oct 30 23:06:00.184837 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:00.184812 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdlsv" Oct 30 23:06:00.184837 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:00.184835 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:06:00.185058 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:00.184901 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xdlsv" podUID="4f7d22b5-256e-47d8-947f-b89b47f14d08" Oct 30 23:06:00.185058 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:00.184942 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:06:00.185058 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:00.185019 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vrpr" podUID="322e3a9b-911f-4cd8-8f43-71d56152e917" Oct 30 23:06:00.185163 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:00.185096 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-htqtp" podUID="dec2c5fa-4b9a-47ed-b005-04874761f419" Oct 30 23:06:00.364391 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:00.364355 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9cltq" event={"ID":"c4368706-761a-4f6e-9dda-a8087a5f4fdc","Type":"ContainerStarted","Data":"98ad294d646b6e008b8ad913e0727911e10b30924969e5ec9424d3f01e169f4b"} Oct 30 23:06:00.585277 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:00.585228 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9cltq" podStartSLOduration=8.612684165 podStartE2EDuration="21.585213167s" totalImagesPullingTime="12.972529002s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:05:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:06:00.53436156 +0000 UTC m=+22.769786349" watchObservedRunningTime="2026-10-30 23:06:00.585213167 +0000 UTC m=+22.820637953" Oct 30 23:06:01.369983 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:01.369816 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 30 23:06:01.370494 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:01.370335 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" event={"ID":"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2","Type":"ContainerStarted","Data":"969ea605beffbf25e06d6626f9bb3fbef7dd46a5e028af8681f6c6f549ed752d"} Oct 30 23:06:01.370853 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:01.370834 2570 scope.go:122] "RemoveContainer" containerID="399eb8749765d7f02db13a1f1488b6b722b03162d5f0119e102148b1ad22f3b7" Oct 30 23:06:02.187498 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:02.187469 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:06:02.187498 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:02.187490 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:06:02.187662 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:02.187590 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vrpr" podUID="322e3a9b-911f-4cd8-8f43-71d56152e917" Oct 30 23:06:02.187662 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:02.187645 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdlsv" Oct 30 23:06:02.187757 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:02.187733 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-htqtp" podUID="dec2c5fa-4b9a-47ed-b005-04874761f419" Oct 30 23:06:02.187845 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:02.187832 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xdlsv" podUID="4f7d22b5-256e-47d8-947f-b89b47f14d08" Oct 30 23:06:02.290863 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:02.290839 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:02.375358 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:02.375337 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 30 23:06:02.375713 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:02.375691 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" event={"ID":"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2","Type":"ContainerStarted","Data":"af9e5ae91d07e03dbcaf46cad2ec21851d8bcd3431f78f5aaba242cc8904d384"} Oct 30 23:06:02.375916 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:02.375903 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:02.375992 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:02.375971 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:02.391120 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:02.391098 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:02.391292 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:02.391278 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:02.633088 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:02.632984 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" podStartSLOduration=8.220177299 podStartE2EDuration="23.632971737s" totalImagesPullingTime="15.412794438s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:05:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:06:02.632579666 +0000 UTC m=+24.868004455" watchObservedRunningTime="2026-10-30 23:06:02.632971737 +0000 UTC m=+24.868396527" Oct 30 23:06:03.229084 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:03.229056 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7vrpr"] Oct 30 23:06:03.229234 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:03.229185 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:06:03.229278 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:03.229261 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vrpr" podUID="322e3a9b-911f-4cd8-8f43-71d56152e917" Oct 30 23:06:03.232088 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:03.232064 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xdlsv"] Oct 30 23:06:03.232188 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:03.232148 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdlsv" Oct 30 23:06:03.232233 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:03.232217 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xdlsv" podUID="4f7d22b5-256e-47d8-947f-b89b47f14d08" Oct 30 23:06:03.234509 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:03.234486 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-htqtp"] Oct 30 23:06:03.234609 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:03.234602 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:06:03.234725 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:03.234705 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-htqtp" podUID="dec2c5fa-4b9a-47ed-b005-04874761f419" Oct 30 23:06:05.184928 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:05.184896 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:06:05.184928 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:05.184936 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdlsv" Oct 30 23:06:05.185516 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:05.184936 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:06:05.185516 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:05.185025 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-htqtp" podUID="dec2c5fa-4b9a-47ed-b005-04874761f419" Oct 30 23:06:05.185516 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:05.185127 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xdlsv" podUID="4f7d22b5-256e-47d8-947f-b89b47f14d08" Oct 30 23:06:05.185516 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:05.185184 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vrpr" podUID="322e3a9b-911f-4cd8-8f43-71d56152e917" Oct 30 23:06:07.184433 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:07.184400 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:06:07.184816 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:07.184490 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdlsv" Oct 30 23:06:07.184816 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:07.184514 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:06:07.184816 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:07.184606 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xdlsv" podUID="4f7d22b5-256e-47d8-947f-b89b47f14d08" Oct 30 23:06:07.184816 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:07.184658 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-htqtp" podUID="dec2c5fa-4b9a-47ed-b005-04874761f419" Oct 30 23:06:07.184816 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:07.184685 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vrpr" podUID="322e3a9b-911f-4cd8-8f43-71d56152e917" Oct 30 23:06:09.049282 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.049060 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeReady" Oct 30 23:06:09.049636 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.049311 2570 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 23:06:09.162887 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.162852 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-szm9n"] Oct 30 23:06:09.167844 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.167828 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-szm9n" Oct 30 23:06:09.184531 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.184513 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdlsv" Oct 30 23:06:09.184624 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.184551 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:06:09.184624 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.184611 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:06:09.213222 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.213197 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-lthcb"] Oct 30 23:06:09.216465 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.216378 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mpbb4"] Oct 30 23:06:09.217975 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.217939 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-lthcb" Oct 30 23:06:09.219459 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.219440 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-9c458c7df-njhmv"] Oct 30 23:06:09.219583 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.219564 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-mpbb4" Oct 30 23:06:09.221818 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.221803 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-dh6nh"] Oct 30 23:06:09.221932 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.221918 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:09.226471 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.226456 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kbrxt"] Oct 30 23:06:09.226577 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.226562 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dh6nh" Oct 30 23:06:09.228879 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.228863 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-kztwm"] Oct 30 23:06:09.229009 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.228991 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kbrxt" Oct 30 23:06:09.231471 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.231452 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5dd6f58df-tctqk"] Oct 30 23:06:09.231552 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.231540 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" Oct 30 23:06:09.234333 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.234317 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chxn6"] Oct 30 23:06:09.234434 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.234418 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:09.236852 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.236836 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5vcm"] Oct 30 23:06:09.236947 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.236934 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chxn6" Oct 30 23:06:09.240661 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.240646 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5vcm" Oct 30 23:06:09.263891 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.263864 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 23:06:09.263891 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.263874 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 23:06:09.264076 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.263991 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 23:06:09.264129 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.264076 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 23:06:09.264129 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.264110 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-tw7s4\"" Oct 30 23:06:09.264217 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.264151 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 23:06:09.264217 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.264158 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-trr47\"" Oct 30 23:06:09.264217 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.264157 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fbzzd\"" Oct 30 23:06:09.264217 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.264191 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f84d4ffd8-wx9nr"] Oct 30 23:06:09.264403 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.264338 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 23:06:09.268951 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.268934 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-g97cr"] Oct 30 23:06:09.269090 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.269075 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:09.272395 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.272373 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-62rf9"] Oct 30 23:06:09.272812 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.272797 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-g97cr" Oct 30 23:06:09.275538 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.275518 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-r26cd"] Oct 30 23:06:09.275706 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.275658 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-62rf9" Oct 30 23:06:09.278138 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.276133 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b754k\" (UniqueName: \"kubernetes.io/projected/67e30e6a-a537-4e2c-a484-f15281828fec-kube-api-access-b754k\") pod \"kas-connection-checker-685895c7c9-szm9n\" (UID: \"67e30e6a-a537-4e2c-a484-f15281828fec\") " pod="kube-system/kas-connection-checker-685895c7c9-szm9n" Oct 30 23:06:09.281576 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.281551 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-j7f9n"] Oct 30 23:06:09.282137 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.282118 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-r26cd" Oct 30 23:06:09.286224 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.286200 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-j9spz"] Oct 30 23:06:09.286505 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.286482 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j7f9n" Oct 30 23:06:09.290706 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.290691 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nzkrw"] Oct 30 23:06:09.290830 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.290815 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j9spz" Oct 30 23:06:09.293484 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.293469 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-szm9n"] Oct 30 23:06:09.293568 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.293495 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-lthcb"] Oct 30 23:06:09.293568 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.293511 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chxn6"] Oct 30 23:06:09.293568 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.293534 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-dh6nh"] Oct 30 23:06:09.293568 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.293556 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kbrxt"] Oct 30 23:06:09.293760 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.293573 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-kztwm"] Oct 30 23:06:09.293760 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.293581 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nzkrw" Oct 30 23:06:09.315119 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.315097 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5dd6f58df-tctqk"] Oct 30 23:06:09.315646 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.315628 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9c458c7df-njhmv"] Oct 30 23:06:09.323012 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.322957 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:06:09.323012 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.322968 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 23:06:09.323364 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.323334 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 23:06:09.323516 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.323499 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 23:06:09.323594 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.323587 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 23:06:09.324644 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.323636 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 23:06:09.324644 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.323657 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 23:06:09.324644 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.323722 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:06:09.324644 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.323735 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:06:09.324644 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.323789 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 23:06:09.324644 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.323755 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:06:09.324644 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.323857 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 23:06:09.324644 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.323907 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 23:06:09.324644 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.323961 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 23:06:09.324644 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.323989 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 23:06:09.324644 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.324001 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 23:06:09.364690 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.364665 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 23:06:09.376186 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.376162 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-r26cd"] Oct 30 23:06:09.376444 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.376425 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h5vcm\" (UID: \"f644db3c-e78a-42a8-9af3-c61194a21839\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5vcm" Oct 30 23:06:09.376520 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.376457 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ec44853-19be-4bbb-9840-bb4c96fd2a29-ca-trust-extracted\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:09.376575 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.376559 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dh6nh\" (UID: \"813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dh6nh" Oct 30 23:06:09.376654 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.376625 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ec44853-19be-4bbb-9840-bb4c96fd2a29-image-registry-private-configuration\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:09.376694 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.376666 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f6c1ad7-10c4-4995-ae18-315d38184360-service-ca-bundle\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:09.376727 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.376694 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-chxn6\" (UID: \"6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chxn6" Oct 30 23:06:09.376765 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.376726 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-tls\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:09.376765 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.376754 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8cbb1813-35e1-49cb-8718-d0ba47d2350d-image-registry-private-configuration\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:09.376835 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.376778 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 23:06:09.376835 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.376781 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvnfz\" (UniqueName: \"kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-kube-api-access-pvnfz\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:09.376897 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.376836 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e9a89b14-a8e8-45f2-84bd-dba21617499c-trusted-ca\") pod \"console-operator-7cd56b8dbc-kztwm\" (UID: \"e9a89b14-a8e8-45f2-84bd-dba21617499c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" Oct 30 23:06:09.376897 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.376883 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 23:06:09.377076 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.376893 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtk5p\" (UniqueName: \"kubernetes.io/projected/8a7f82b8-115d-43ed-a7a2-cb869a7e48e0-kube-api-access-rtk5p\") pod \"kas-connection-checker-685895c7c9-lthcb\" (UID: \"8a7f82b8-115d-43ed-a7a2-cb869a7e48e0\") " pod="kube-system/kas-connection-checker-685895c7c9-lthcb" Oct 30 23:06:09.377076 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.376937 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-metrics-certs\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:09.377076 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.376964 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-bound-sa-token\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:09.377076 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.377019 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 23:06:09.377339 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.377083 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 23:06:09.377339 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.377083 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ec44853-19be-4bbb-9840-bb4c96fd2a29-installation-pull-secrets\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:09.377339 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.377114 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 23:06:09.377339 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.377136 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8cbb1813-35e1-49cb-8718-d0ba47d2350d-installation-pull-secrets\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:09.377339 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.377183 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-snapshots\") pod \"insights-operator-7c4986d449-g97cr\" (UID: \"ef598d5f-a066-4b2d-96bd-fbfc47741dbb\") " pod="openshift-insights/insights-operator-7c4986d449-g97cr" Oct 30 23:06:09.377339 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.377187 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 23:06:09.377339 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.377248 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-dh6nh\" (UID: \"813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dh6nh" Oct 30 23:06:09.377339 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.377284 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 23:06:09.377339 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.377322 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-certificates\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:09.377339 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.377336 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 23:06:09.377826 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.377359 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-tmp\") pod \"insights-operator-7c4986d449-g97cr\" (UID: \"ef598d5f-a066-4b2d-96bd-fbfc47741dbb\") " pod="openshift-insights/insights-operator-7c4986d449-g97cr" Oct 30 23:06:09.377826 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.377388 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 23:06:09.377826 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.377402 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-service-ca-bundle\") pod \"insights-operator-7c4986d449-g97cr\" (UID: \"ef598d5f-a066-4b2d-96bd-fbfc47741dbb\") " pod="openshift-insights/insights-operator-7c4986d449-g97cr" Oct 30 23:06:09.377826 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.377447 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 23:06:09.377826 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.377483 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 23:06:09.377826 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.377551 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 23:06:09.377826 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.377582 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 23:06:09.377826 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.377547 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8cbb1813-35e1-49cb-8718-d0ba47d2350d-trusted-ca\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:09.377826 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.377627 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 23:06:09.377826 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.377647 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-tls\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:09.377826 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.377674 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-g97cr\" (UID: \"ef598d5f-a066-4b2d-96bd-fbfc47741dbb\") " pod="openshift-insights/insights-operator-7c4986d449-g97cr" Oct 30 23:06:09.377826 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.377708 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-serving-cert\") pod \"insights-operator-7c4986d449-g97cr\" (UID: \"ef598d5f-a066-4b2d-96bd-fbfc47741dbb\") " pod="openshift-insights/insights-operator-7c4986d449-g97cr" Oct 30 23:06:09.377826 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.377757 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e9a89b14-a8e8-45f2-84bd-dba21617499c-config\") pod \"console-operator-7cd56b8dbc-kztwm\" (UID: \"e9a89b14-a8e8-45f2-84bd-dba21617499c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" Oct 30 23:06:09.377826 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.377785 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc7nh\" (UniqueName: \"kubernetes.io/projected/e9a89b14-a8e8-45f2-84bd-dba21617499c-kube-api-access-fc7nh\") pod \"console-operator-7cd56b8dbc-kztwm\" (UID: \"e9a89b14-a8e8-45f2-84bd-dba21617499c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" Oct 30 23:06:09.378562 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.377843 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-stats-auth\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:09.378562 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.377869 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-certificates\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:09.378562 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.377906 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-chxn6\" (UID: \"6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chxn6" Oct 30 23:06:09.378562 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.377949 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b754k\" (UniqueName: \"kubernetes.io/projected/67e30e6a-a537-4e2c-a484-f15281828fec-kube-api-access-b754k\") pod \"kas-connection-checker-685895c7c9-szm9n\" (UID: \"67e30e6a-a537-4e2c-a484-f15281828fec\") " pod="kube-system/kas-connection-checker-685895c7c9-szm9n" Oct 30 23:06:09.378562 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.377994 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-bound-sa-token\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:09.378562 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.378025 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbm49\" (UniqueName: \"kubernetes.io/projected/f644db3c-e78a-42a8-9af3-c61194a21839-kube-api-access-qbm49\") pod \"cluster-samples-operator-6ddb6c8b94-h5vcm\" (UID: \"f644db3c-e78a-42a8-9af3-c61194a21839\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5vcm" Oct 30 23:06:09.378562 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.378070 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldxjv\" (UniqueName: \"kubernetes.io/projected/ad111ef6-4d18-4438-a0cc-e31a46804710-kube-api-access-ldxjv\") pod \"network-check-source-6f78d59f99-62rf9\" (UID: \"ad111ef6-4d18-4438-a0cc-e31a46804710\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-62rf9" Oct 30 23:06:09.378562 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.378096 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp7ls\" (UniqueName: \"kubernetes.io/projected/e32c5dc1-43ac-4ca5-a391-947b68d10d51-kube-api-access-xp7ls\") pod \"volume-data-source-validator-58c9f5f46d-kbrxt\" (UID: \"e32c5dc1-43ac-4ca5-a391-947b68d10d51\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kbrxt" Oct 30 23:06:09.378562 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.378125 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8czng\" (UniqueName: \"kubernetes.io/projected/5f6c1ad7-10c4-4995-ae18-315d38184360-kube-api-access-8czng\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:09.378562 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.378149 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e9a89b14-a8e8-45f2-84bd-dba21617499c-serving-cert\") pod \"console-operator-7cd56b8dbc-kztwm\" (UID: \"e9a89b14-a8e8-45f2-84bd-dba21617499c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" Oct 30 23:06:09.378562 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.378171 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0cb2a164-a8b5-4b93-b656-304550f04179-nginx-conf\") pod \"networking-console-plugin-86c478bb88-r26cd\" (UID: \"0cb2a164-a8b5-4b93-b656-304550f04179\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r26cd" Oct 30 23:06:09.378562 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.378196 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcls9\" (UniqueName: \"kubernetes.io/projected/624d7fe2-4960-48ba-b711-3d85e4bf0687-kube-api-access-gcls9\") pod \"kas-connection-checker-685895c7c9-mpbb4\" (UID: \"624d7fe2-4960-48ba-b711-3d85e4bf0687\") " pod="kube-system/kas-connection-checker-685895c7c9-mpbb4" Oct 30 23:06:09.378562 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.378221 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-h5vcm\" (UID: \"f644db3c-e78a-42a8-9af3-c61194a21839\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5vcm" Oct 30 23:06:09.378562 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.378248 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ec44853-19be-4bbb-9840-bb4c96fd2a29-trusted-ca\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:09.378562 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.378270 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpjxg\" (UniqueName: \"kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-kube-api-access-tpjxg\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:09.379254 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.378299 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvh4w\" (UniqueName: \"kubernetes.io/projected/6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893-kube-api-access-vvh4w\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-chxn6\" (UID: \"6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chxn6" Oct 30 23:06:09.379254 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.378368 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-g97cr"] Oct 30 23:06:09.379254 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.378390 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8cbb1813-35e1-49cb-8718-d0ba47d2350d-ca-trust-extracted\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:09.379254 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.378505 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-2fmdc\"" Oct 30 23:06:09.379254 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.378556 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl67b\" (UniqueName: \"kubernetes.io/projected/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-kube-api-access-kl67b\") pod \"insights-operator-7c4986d449-g97cr\" (UID: \"ef598d5f-a066-4b2d-96bd-fbfc47741dbb\") " pod="openshift-insights/insights-operator-7c4986d449-g97cr" Oct 30 23:06:09.379254 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.378585 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cl4w9\"" Oct 30 23:06:09.379254 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.378663 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cqcdm\"" Oct 30 23:06:09.379254 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.378634 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g48ld\" (UniqueName: \"kubernetes.io/projected/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-kube-api-access-g48ld\") pod \"cluster-monitoring-operator-884576c57-dh6nh\" (UID: \"813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dh6nh" Oct 30 23:06:09.379254 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.378715 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0cb2a164-a8b5-4b93-b656-304550f04179-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-r26cd\" (UID: \"0cb2a164-a8b5-4b93-b656-304550f04179\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r26cd" Oct 30 23:06:09.379254 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.378722 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 23:06:09.379254 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.378753 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 23:06:09.379254 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.378797 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-l6bwx\"" Oct 30 23:06:09.379254 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.378807 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-default-certificate\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:09.379254 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.378892 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 23:06:09.379254 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.379255 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mpbb4"] Oct 30 23:06:09.380200 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.380183 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nzkrw"] Oct 30 23:06:09.381348 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.381332 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-j9spz"] Oct 30 23:06:09.381965 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.381952 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5vcm"] Oct 30 23:06:09.382560 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.382546 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-j7f9n"] Oct 30 23:06:09.383156 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.383137 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f84d4ffd8-wx9nr"] Oct 30 23:06:09.389621 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.389596 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-62rf9"] Oct 30 23:06:09.423011 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.422985 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 23:06:09.423011 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.422985 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 23:06:09.423378 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.423149 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:06:09.423378 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.423177 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 23:06:09.423378 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.423226 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 23:06:09.423378 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.423249 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 23:06:09.423378 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.423271 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 23:06:09.423378 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.423298 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 23:06:09.423378 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.423321 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 23:06:09.423378 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.423355 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 23:06:09.427827 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.425008 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 23:06:09.436961 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.436933 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b754k\" (UniqueName: \"kubernetes.io/projected/67e30e6a-a537-4e2c-a484-f15281828fec-kube-api-access-b754k\") pod \"kas-connection-checker-685895c7c9-szm9n\" (UID: \"67e30e6a-a537-4e2c-a484-f15281828fec\") " pod="kube-system/kas-connection-checker-685895c7c9-szm9n" Oct 30 23:06:09.479769 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.479747 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-szm9n" Oct 30 23:06:09.479905 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.479881 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ec44853-19be-4bbb-9840-bb4c96fd2a29-installation-pull-secrets\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:09.479965 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.479922 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8cbb1813-35e1-49cb-8718-d0ba47d2350d-installation-pull-secrets\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:09.479965 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.479951 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-snapshots\") pod \"insights-operator-7c4986d449-g97cr\" (UID: \"ef598d5f-a066-4b2d-96bd-fbfc47741dbb\") " pod="openshift-insights/insights-operator-7c4986d449-g97cr" Oct 30 23:06:09.480102 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.479974 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-dh6nh\" (UID: \"813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dh6nh" Oct 30 23:06:09.480102 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.480001 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-certificates\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:09.480102 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.480026 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-tmp\") pod \"insights-operator-7c4986d449-g97cr\" (UID: \"ef598d5f-a066-4b2d-96bd-fbfc47741dbb\") " pod="openshift-insights/insights-operator-7c4986d449-g97cr" Oct 30 23:06:09.480102 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.480073 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-service-ca-bundle\") pod \"insights-operator-7c4986d449-g97cr\" (UID: \"ef598d5f-a066-4b2d-96bd-fbfc47741dbb\") " pod="openshift-insights/insights-operator-7c4986d449-g97cr" Oct 30 23:06:09.480282 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.480104 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg5n4\" (UniqueName: \"kubernetes.io/projected/582d0663-6eba-4e69-81b0-fb25380f4c47-kube-api-access-vg5n4\") pod \"ingress-canary-nzkrw\" (UID: \"582d0663-6eba-4e69-81b0-fb25380f4c47\") " pod="openshift-ingress-canary/ingress-canary-nzkrw" Oct 30 23:06:09.480282 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.480132 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8cbb1813-35e1-49cb-8718-d0ba47d2350d-trusted-ca\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:09.480282 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.480159 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-tls\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:09.480282 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.480185 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-g97cr\" (UID: \"ef598d5f-a066-4b2d-96bd-fbfc47741dbb\") " pod="openshift-insights/insights-operator-7c4986d449-g97cr" Oct 30 23:06:09.480282 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.480208 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-serving-cert\") pod \"insights-operator-7c4986d449-g97cr\" (UID: \"ef598d5f-a066-4b2d-96bd-fbfc47741dbb\") " pod="openshift-insights/insights-operator-7c4986d449-g97cr" Oct 30 23:06:09.480282 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.480234 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e9a89b14-a8e8-45f2-84bd-dba21617499c-config\") pod \"console-operator-7cd56b8dbc-kztwm\" (UID: \"e9a89b14-a8e8-45f2-84bd-dba21617499c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" Oct 30 23:06:09.480282 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.480258 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc7nh\" (UniqueName: \"kubernetes.io/projected/e9a89b14-a8e8-45f2-84bd-dba21617499c-kube-api-access-fc7nh\") pod \"console-operator-7cd56b8dbc-kztwm\" (UID: \"e9a89b14-a8e8-45f2-84bd-dba21617499c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" Oct 30 23:06:09.480527 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.480289 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-stats-auth\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:09.480527 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.480319 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-certificates\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:09.480527 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.480346 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-chxn6\" (UID: \"6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chxn6" Oct 30 23:06:09.480527 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.480381 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-bound-sa-token\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:09.480527 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.480413 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbm49\" (UniqueName: \"kubernetes.io/projected/f644db3c-e78a-42a8-9af3-c61194a21839-kube-api-access-qbm49\") pod \"cluster-samples-operator-6ddb6c8b94-h5vcm\" (UID: \"f644db3c-e78a-42a8-9af3-c61194a21839\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5vcm" Oct 30 23:06:09.480527 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.480438 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldxjv\" (UniqueName: \"kubernetes.io/projected/ad111ef6-4d18-4438-a0cc-e31a46804710-kube-api-access-ldxjv\") pod \"network-check-source-6f78d59f99-62rf9\" (UID: \"ad111ef6-4d18-4438-a0cc-e31a46804710\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-62rf9" Oct 30 23:06:09.480527 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.480464 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp7ls\" (UniqueName: \"kubernetes.io/projected/e32c5dc1-43ac-4ca5-a391-947b68d10d51-kube-api-access-xp7ls\") pod \"volume-data-source-validator-58c9f5f46d-kbrxt\" (UID: \"e32c5dc1-43ac-4ca5-a391-947b68d10d51\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kbrxt" Oct 30 23:06:09.480527 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.480494 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58d18552-3df9-47f4-9291-b0e82cd954d2-metrics-tls\") pod \"dns-default-j9spz\" (UID: \"58d18552-3df9-47f4-9291-b0e82cd954d2\") " pod="openshift-dns/dns-default-j9spz" Oct 30 23:06:09.480527 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.480519 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58d18552-3df9-47f4-9291-b0e82cd954d2-tmp-dir\") pod \"dns-default-j9spz\" (UID: \"58d18552-3df9-47f4-9291-b0e82cd954d2\") " pod="openshift-dns/dns-default-j9spz" Oct 30 23:06:09.480936 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.480520 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-tmp\") pod \"insights-operator-7c4986d449-g97cr\" (UID: \"ef598d5f-a066-4b2d-96bd-fbfc47741dbb\") " pod="openshift-insights/insights-operator-7c4986d449-g97cr" Oct 30 23:06:09.480936 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.480664 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8czng\" (UniqueName: \"kubernetes.io/projected/5f6c1ad7-10c4-4995-ae18-315d38184360-kube-api-access-8czng\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:09.480936 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:09.480907 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:06:09.480936 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:09.480925 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f84d4ffd8-wx9nr: secret "image-registry-tls" not found Oct 30 23:06:09.481162 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:09.480991 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-tls podName:8cbb1813-35e1-49cb-8718-d0ba47d2350d nodeName:}" failed. No retries permitted until 2026-10-30 23:06:09.980971727 +0000 UTC m=+32.216396496 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-tls") pod "image-registry-6f84d4ffd8-wx9nr" (UID: "8cbb1813-35e1-49cb-8718-d0ba47d2350d") : secret "image-registry-tls" not found Oct 30 23:06:09.481162 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.480919 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e9a89b14-a8e8-45f2-84bd-dba21617499c-serving-cert\") pod \"console-operator-7cd56b8dbc-kztwm\" (UID: \"e9a89b14-a8e8-45f2-84bd-dba21617499c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" Oct 30 23:06:09.481162 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.481148 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/582d0663-6eba-4e69-81b0-fb25380f4c47-cert\") pod \"ingress-canary-nzkrw\" (UID: \"582d0663-6eba-4e69-81b0-fb25380f4c47\") " pod="openshift-ingress-canary/ingress-canary-nzkrw" Oct 30 23:06:09.481162 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.481155 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-service-ca-bundle\") pod \"insights-operator-7c4986d449-g97cr\" (UID: \"ef598d5f-a066-4b2d-96bd-fbfc47741dbb\") " pod="openshift-insights/insights-operator-7c4986d449-g97cr" Oct 30 23:06:09.481372 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.481184 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0cb2a164-a8b5-4b93-b656-304550f04179-nginx-conf\") pod \"networking-console-plugin-86c478bb88-r26cd\" (UID: \"0cb2a164-a8b5-4b93-b656-304550f04179\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r26cd" Oct 30 23:06:09.481372 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.481228 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcls9\" (UniqueName: \"kubernetes.io/projected/624d7fe2-4960-48ba-b711-3d85e4bf0687-kube-api-access-gcls9\") pod \"kas-connection-checker-685895c7c9-mpbb4\" (UID: \"624d7fe2-4960-48ba-b711-3d85e4bf0687\") " pod="kube-system/kas-connection-checker-685895c7c9-mpbb4" Oct 30 23:06:09.481372 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.481256 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-h5vcm\" (UID: \"f644db3c-e78a-42a8-9af3-c61194a21839\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5vcm" Oct 30 23:06:09.481372 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.481289 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvlmc\" (UniqueName: \"kubernetes.io/projected/58d18552-3df9-47f4-9291-b0e82cd954d2-kube-api-access-zvlmc\") pod \"dns-default-j9spz\" (UID: \"58d18552-3df9-47f4-9291-b0e82cd954d2\") " pod="openshift-dns/dns-default-j9spz" Oct 30 23:06:09.481372 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.481318 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ec44853-19be-4bbb-9840-bb4c96fd2a29-trusted-ca\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:09.481372 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.481349 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpjxg\" (UniqueName: \"kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-kube-api-access-tpjxg\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:09.481655 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.481378 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvh4w\" (UniqueName: \"kubernetes.io/projected/6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893-kube-api-access-vvh4w\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-chxn6\" (UID: \"6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chxn6" Oct 30 23:06:09.481655 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.481406 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8cbb1813-35e1-49cb-8718-d0ba47d2350d-ca-trust-extracted\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:09.481655 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.481503 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl67b\" (UniqueName: \"kubernetes.io/projected/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-kube-api-access-kl67b\") pod \"insights-operator-7c4986d449-g97cr\" (UID: \"ef598d5f-a066-4b2d-96bd-fbfc47741dbb\") " pod="openshift-insights/insights-operator-7c4986d449-g97cr" Oct 30 23:06:09.481655 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.481534 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58d18552-3df9-47f4-9291-b0e82cd954d2-config-volume\") pod \"dns-default-j9spz\" (UID: \"58d18552-3df9-47f4-9291-b0e82cd954d2\") " pod="openshift-dns/dns-default-j9spz" Oct 30 23:06:09.481655 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.481580 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g48ld\" (UniqueName: \"kubernetes.io/projected/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-kube-api-access-g48ld\") pod \"cluster-monitoring-operator-884576c57-dh6nh\" (UID: \"813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dh6nh" Oct 30 23:06:09.481655 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.481610 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0cb2a164-a8b5-4b93-b656-304550f04179-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-r26cd\" (UID: \"0cb2a164-a8b5-4b93-b656-304550f04179\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r26cd" Oct 30 23:06:09.481931 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.481655 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-default-certificate\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:09.481931 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.481718 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h5vcm\" (UID: \"f644db3c-e78a-42a8-9af3-c61194a21839\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5vcm" Oct 30 23:06:09.481931 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.481746 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/edc2af65-1515-4742-89e1-b49814016b09-serving-cert\") pod \"service-ca-operator-544d749c8c-j7f9n\" (UID: \"edc2af65-1515-4742-89e1-b49814016b09\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j7f9n" Oct 30 23:06:09.481931 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.481779 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ec44853-19be-4bbb-9840-bb4c96fd2a29-ca-trust-extracted\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:09.481931 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.481810 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dh6nh\" (UID: \"813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dh6nh" Oct 30 23:06:09.481931 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.481867 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ec44853-19be-4bbb-9840-bb4c96fd2a29-image-registry-private-configuration\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:09.481931 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.481904 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f6c1ad7-10c4-4995-ae18-315d38184360-service-ca-bundle\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:09.482284 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.481932 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-chxn6\" (UID: \"6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chxn6" Oct 30 23:06:09.482284 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.481962 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-tls\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:09.482284 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.481993 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8cbb1813-35e1-49cb-8718-d0ba47d2350d-image-registry-private-configuration\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:09.482284 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.482028 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvnfz\" (UniqueName: \"kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-kube-api-access-pvnfz\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:09.482284 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.482076 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e9a89b14-a8e8-45f2-84bd-dba21617499c-trusted-ca\") pod \"console-operator-7cd56b8dbc-kztwm\" (UID: \"e9a89b14-a8e8-45f2-84bd-dba21617499c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" Oct 30 23:06:09.482284 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.482104 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/edc2af65-1515-4742-89e1-b49814016b09-config\") pod \"service-ca-operator-544d749c8c-j7f9n\" (UID: \"edc2af65-1515-4742-89e1-b49814016b09\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j7f9n" Oct 30 23:06:09.482284 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.482139 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtk5p\" (UniqueName: \"kubernetes.io/projected/8a7f82b8-115d-43ed-a7a2-cb869a7e48e0-kube-api-access-rtk5p\") pod \"kas-connection-checker-685895c7c9-lthcb\" (UID: \"8a7f82b8-115d-43ed-a7a2-cb869a7e48e0\") " pod="kube-system/kas-connection-checker-685895c7c9-lthcb" Oct 30 23:06:09.482284 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.482167 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-metrics-certs\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:09.482284 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.482204 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-bound-sa-token\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:09.482284 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.482239 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bxrl\" (UniqueName: \"kubernetes.io/projected/edc2af65-1515-4742-89e1-b49814016b09-kube-api-access-6bxrl\") pod \"service-ca-operator-544d749c8c-j7f9n\" (UID: \"edc2af65-1515-4742-89e1-b49814016b09\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j7f9n" Oct 30 23:06:09.482696 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:09.482314 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 23:06:09.482696 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:09.482363 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cb2a164-a8b5-4b93-b656-304550f04179-networking-console-plugin-cert podName:0cb2a164-a8b5-4b93-b656-304550f04179 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:09.982347792 +0000 UTC m=+32.217772560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0cb2a164-a8b5-4b93-b656-304550f04179-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-r26cd" (UID: "0cb2a164-a8b5-4b93-b656-304550f04179") : secret "networking-console-plugin-cert" not found Oct 30 23:06:09.482808 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:09.482797 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:06:09.482860 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:09.482812 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9c458c7df-njhmv: secret "image-registry-tls" not found Oct 30 23:06:09.482860 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:09.482855 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-tls podName:3ec44853-19be-4bbb-9840-bb4c96fd2a29 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:09.982840732 +0000 UTC m=+32.218265525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-tls") pod "image-registry-9c458c7df-njhmv" (UID: "3ec44853-19be-4bbb-9840-bb4c96fd2a29") : secret "image-registry-tls" not found Oct 30 23:06:09.482969 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:09.482923 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 23:06:09.482969 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.482918 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-certificates\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:09.482969 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:09.482960 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-tls podName:f644db3c-e78a-42a8-9af3-c61194a21839 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:09.982949429 +0000 UTC m=+32.218374225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h5vcm" (UID: "f644db3c-e78a-42a8-9af3-c61194a21839") : secret "samples-operator-tls" not found Oct 30 23:06:09.483191 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.483166 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ec44853-19be-4bbb-9840-bb4c96fd2a29-installation-pull-secrets\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:09.483366 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.483344 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ec44853-19be-4bbb-9840-bb4c96fd2a29-ca-trust-extracted\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:09.483441 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.483370 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-g97cr\" (UID: \"ef598d5f-a066-4b2d-96bd-fbfc47741dbb\") " pod="openshift-insights/insights-operator-7c4986d449-g97cr" Oct 30 23:06:09.483441 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.483398 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-dh6nh\" (UID: \"813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dh6nh" Oct 30 23:06:09.483522 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:09.483445 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 23:06:09.483522 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:09.483487 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-cluster-monitoring-operator-tls podName:813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:09.983474319 +0000 UTC m=+32.218899088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-dh6nh" (UID: "813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5") : secret "cluster-monitoring-operator-tls" not found Oct 30 23:06:09.483860 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:09.483717 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f6c1ad7-10c4-4995-ae18-315d38184360-service-ca-bundle podName:5f6c1ad7-10c4-4995-ae18-315d38184360 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:09.983699722 +0000 UTC m=+32.219124517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5f6c1ad7-10c4-4995-ae18-315d38184360-service-ca-bundle") pod "router-default-5dd6f58df-tctqk" (UID: "5f6c1ad7-10c4-4995-ae18-315d38184360") : configmap references non-existent config key: service-ca.crt Oct 30 23:06:09.484076 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.483972 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e9a89b14-a8e8-45f2-84bd-dba21617499c-config\") pod \"console-operator-7cd56b8dbc-kztwm\" (UID: \"e9a89b14-a8e8-45f2-84bd-dba21617499c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" Oct 30 23:06:09.484171 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:09.484140 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 23:06:09.484227 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:09.484190 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-metrics-certs podName:5f6c1ad7-10c4-4995-ae18-315d38184360 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:09.984174629 +0000 UTC m=+32.219599402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-metrics-certs") pod "router-default-5dd6f58df-tctqk" (UID: "5f6c1ad7-10c4-4995-ae18-315d38184360") : secret "router-metrics-certs-default" not found Oct 30 23:06:09.484292 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.484253 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-stats-auth\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:09.484922 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.484592 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0cb2a164-a8b5-4b93-b656-304550f04179-nginx-conf\") pod \"networking-console-plugin-86c478bb88-r26cd\" (UID: \"0cb2a164-a8b5-4b93-b656-304550f04179\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r26cd" Oct 30 23:06:09.485295 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.481536 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-chxn6\" (UID: \"6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chxn6" Oct 30 23:06:09.485295 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.485170 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e9a89b14-a8e8-45f2-84bd-dba21617499c-trusted-ca\") pod \"console-operator-7cd56b8dbc-kztwm\" (UID: \"e9a89b14-a8e8-45f2-84bd-dba21617499c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" Oct 30 23:06:09.485295 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.485213 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-h5vcm\" (UID: \"f644db3c-e78a-42a8-9af3-c61194a21839\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5vcm" Oct 30 23:06:09.486213 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.486175 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ec44853-19be-4bbb-9840-bb4c96fd2a29-trusted-ca\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:09.486347 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.486322 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ec44853-19be-4bbb-9840-bb4c96fd2a29-image-registry-private-configuration\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:09.487156 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.487134 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-serving-cert\") pod \"insights-operator-7c4986d449-g97cr\" (UID: \"ef598d5f-a066-4b2d-96bd-fbfc47741dbb\") " pod="openshift-insights/insights-operator-7c4986d449-g97cr" Oct 30 23:06:09.487376 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.487335 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-chxn6\" (UID: \"6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chxn6" Oct 30 23:06:09.487452 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.487412 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-snapshots\") pod \"insights-operator-7c4986d449-g97cr\" (UID: \"ef598d5f-a066-4b2d-96bd-fbfc47741dbb\") " pod="openshift-insights/insights-operator-7c4986d449-g97cr" Oct 30 23:06:09.487695 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.487670 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8cbb1813-35e1-49cb-8718-d0ba47d2350d-installation-pull-secrets\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:09.487770 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.487695 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8cbb1813-35e1-49cb-8718-d0ba47d2350d-ca-trust-extracted\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:09.487879 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.487859 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-certificates\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:09.487983 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.487966 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8cbb1813-35e1-49cb-8718-d0ba47d2350d-image-registry-private-configuration\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:09.488078 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.488005 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8cbb1813-35e1-49cb-8718-d0ba47d2350d-trusted-ca\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:09.488134 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.488084 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e9a89b14-a8e8-45f2-84bd-dba21617499c-serving-cert\") pod \"console-operator-7cd56b8dbc-kztwm\" (UID: \"e9a89b14-a8e8-45f2-84bd-dba21617499c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" Oct 30 23:06:09.489849 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.489773 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-default-certificate\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:09.541796 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.541558 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldxjv\" (UniqueName: \"kubernetes.io/projected/ad111ef6-4d18-4438-a0cc-e31a46804710-kube-api-access-ldxjv\") pod \"network-check-source-6f78d59f99-62rf9\" (UID: \"ad111ef6-4d18-4438-a0cc-e31a46804710\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-62rf9" Oct 30 23:06:09.541942 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.541772 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc7nh\" (UniqueName: \"kubernetes.io/projected/e9a89b14-a8e8-45f2-84bd-dba21617499c-kube-api-access-fc7nh\") pod \"console-operator-7cd56b8dbc-kztwm\" (UID: \"e9a89b14-a8e8-45f2-84bd-dba21617499c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" Oct 30 23:06:09.545604 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.545457 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g48ld\" (UniqueName: \"kubernetes.io/projected/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-kube-api-access-g48ld\") pod \"cluster-monitoring-operator-884576c57-dh6nh\" (UID: \"813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dh6nh" Oct 30 23:06:09.545911 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.545758 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtk5p\" (UniqueName: \"kubernetes.io/projected/8a7f82b8-115d-43ed-a7a2-cb869a7e48e0-kube-api-access-rtk5p\") pod \"kas-connection-checker-685895c7c9-lthcb\" (UID: \"8a7f82b8-115d-43ed-a7a2-cb869a7e48e0\") " pod="kube-system/kas-connection-checker-685895c7c9-lthcb" Oct 30 23:06:09.545911 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.545852 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8czng\" (UniqueName: \"kubernetes.io/projected/5f6c1ad7-10c4-4995-ae18-315d38184360-kube-api-access-8czng\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:09.545911 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.545868 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcls9\" (UniqueName: \"kubernetes.io/projected/624d7fe2-4960-48ba-b711-3d85e4bf0687-kube-api-access-gcls9\") pod \"kas-connection-checker-685895c7c9-mpbb4\" (UID: \"624d7fe2-4960-48ba-b711-3d85e4bf0687\") " pod="kube-system/kas-connection-checker-685895c7c9-mpbb4" Oct 30 23:06:09.546124 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.546071 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpjxg\" (UniqueName: \"kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-kube-api-access-tpjxg\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:09.546586 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.546564 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-bound-sa-token\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:09.546674 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.546655 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbm49\" (UniqueName: \"kubernetes.io/projected/f644db3c-e78a-42a8-9af3-c61194a21839-kube-api-access-qbm49\") pod \"cluster-samples-operator-6ddb6c8b94-h5vcm\" (UID: \"f644db3c-e78a-42a8-9af3-c61194a21839\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5vcm" Oct 30 23:06:09.546850 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.546767 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl67b\" (UniqueName: \"kubernetes.io/projected/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-kube-api-access-kl67b\") pod \"insights-operator-7c4986d449-g97cr\" (UID: \"ef598d5f-a066-4b2d-96bd-fbfc47741dbb\") " pod="openshift-insights/insights-operator-7c4986d449-g97cr" Oct 30 23:06:09.546850 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.546799 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvh4w\" (UniqueName: \"kubernetes.io/projected/6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893-kube-api-access-vvh4w\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-chxn6\" (UID: \"6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chxn6" Oct 30 23:06:09.547361 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.547341 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvnfz\" (UniqueName: \"kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-kube-api-access-pvnfz\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:09.547793 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.547735 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-bound-sa-token\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:09.548557 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.548521 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp7ls\" (UniqueName: \"kubernetes.io/projected/e32c5dc1-43ac-4ca5-a391-947b68d10d51-kube-api-access-xp7ls\") pod \"volume-data-source-validator-58c9f5f46d-kbrxt\" (UID: \"e32c5dc1-43ac-4ca5-a391-947b68d10d51\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kbrxt" Oct 30 23:06:09.555424 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.555385 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kbrxt" Oct 30 23:06:09.562544 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.562526 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" Oct 30 23:06:09.577791 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.577773 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chxn6" Oct 30 23:06:09.583136 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.583115 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/582d0663-6eba-4e69-81b0-fb25380f4c47-cert\") pod \"ingress-canary-nzkrw\" (UID: \"582d0663-6eba-4e69-81b0-fb25380f4c47\") " pod="openshift-ingress-canary/ingress-canary-nzkrw" Oct 30 23:06:09.583230 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.583146 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvlmc\" (UniqueName: \"kubernetes.io/projected/58d18552-3df9-47f4-9291-b0e82cd954d2-kube-api-access-zvlmc\") pod \"dns-default-j9spz\" (UID: \"58d18552-3df9-47f4-9291-b0e82cd954d2\") " pod="openshift-dns/dns-default-j9spz" Oct 30 23:06:09.583230 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.583178 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58d18552-3df9-47f4-9291-b0e82cd954d2-config-volume\") pod \"dns-default-j9spz\" (UID: \"58d18552-3df9-47f4-9291-b0e82cd954d2\") " pod="openshift-dns/dns-default-j9spz" Oct 30 23:06:09.583574 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.583423 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/edc2af65-1515-4742-89e1-b49814016b09-serving-cert\") pod \"service-ca-operator-544d749c8c-j7f9n\" (UID: \"edc2af65-1515-4742-89e1-b49814016b09\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j7f9n" Oct 30 23:06:09.583574 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.583525 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/edc2af65-1515-4742-89e1-b49814016b09-config\") pod \"service-ca-operator-544d749c8c-j7f9n\" (UID: \"edc2af65-1515-4742-89e1-b49814016b09\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j7f9n" Oct 30 23:06:09.583574 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.583570 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bxrl\" (UniqueName: \"kubernetes.io/projected/edc2af65-1515-4742-89e1-b49814016b09-kube-api-access-6bxrl\") pod \"service-ca-operator-544d749c8c-j7f9n\" (UID: \"edc2af65-1515-4742-89e1-b49814016b09\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j7f9n" Oct 30 23:06:09.583763 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.583623 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg5n4\" (UniqueName: \"kubernetes.io/projected/582d0663-6eba-4e69-81b0-fb25380f4c47-kube-api-access-vg5n4\") pod \"ingress-canary-nzkrw\" (UID: \"582d0663-6eba-4e69-81b0-fb25380f4c47\") " pod="openshift-ingress-canary/ingress-canary-nzkrw" Oct 30 23:06:09.583763 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.583688 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58d18552-3df9-47f4-9291-b0e82cd954d2-metrics-tls\") pod \"dns-default-j9spz\" (UID: \"58d18552-3df9-47f4-9291-b0e82cd954d2\") " pod="openshift-dns/dns-default-j9spz" Oct 30 23:06:09.583763 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.583713 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58d18552-3df9-47f4-9291-b0e82cd954d2-config-volume\") pod \"dns-default-j9spz\" (UID: \"58d18552-3df9-47f4-9291-b0e82cd954d2\") " pod="openshift-dns/dns-default-j9spz" Oct 30 23:06:09.583763 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.583719 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58d18552-3df9-47f4-9291-b0e82cd954d2-tmp-dir\") pod \"dns-default-j9spz\" (UID: \"58d18552-3df9-47f4-9291-b0e82cd954d2\") " pod="openshift-dns/dns-default-j9spz" Oct 30 23:06:09.583945 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:09.583850 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 23:06:09.583945 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:09.583913 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58d18552-3df9-47f4-9291-b0e82cd954d2-metrics-tls podName:58d18552-3df9-47f4-9291-b0e82cd954d2 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:10.083896063 +0000 UTC m=+32.319320835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/58d18552-3df9-47f4-9291-b0e82cd954d2-metrics-tls") pod "dns-default-j9spz" (UID: "58d18552-3df9-47f4-9291-b0e82cd954d2") : secret "dns-default-metrics-tls" not found Oct 30 23:06:09.584097 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.583962 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58d18552-3df9-47f4-9291-b0e82cd954d2-tmp-dir\") pod \"dns-default-j9spz\" (UID: \"58d18552-3df9-47f4-9291-b0e82cd954d2\") " pod="openshift-dns/dns-default-j9spz" Oct 30 23:06:09.584097 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.584065 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/edc2af65-1515-4742-89e1-b49814016b09-config\") pod \"service-ca-operator-544d749c8c-j7f9n\" (UID: \"edc2af65-1515-4742-89e1-b49814016b09\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j7f9n" Oct 30 23:06:09.585541 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.585522 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/582d0663-6eba-4e69-81b0-fb25380f4c47-cert\") pod \"ingress-canary-nzkrw\" (UID: \"582d0663-6eba-4e69-81b0-fb25380f4c47\") " pod="openshift-ingress-canary/ingress-canary-nzkrw" Oct 30 23:06:09.585710 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.585694 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/edc2af65-1515-4742-89e1-b49814016b09-serving-cert\") pod \"service-ca-operator-544d749c8c-j7f9n\" (UID: \"edc2af65-1515-4742-89e1-b49814016b09\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j7f9n" Oct 30 23:06:09.601287 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.600433 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-g97cr" Oct 30 23:06:09.606801 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.606773 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-62rf9" Oct 30 23:06:09.641095 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.641069 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvlmc\" (UniqueName: \"kubernetes.io/projected/58d18552-3df9-47f4-9291-b0e82cd954d2-kube-api-access-zvlmc\") pod \"dns-default-j9spz\" (UID: \"58d18552-3df9-47f4-9291-b0e82cd954d2\") " pod="openshift-dns/dns-default-j9spz" Oct 30 23:06:09.642424 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.642357 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bxrl\" (UniqueName: \"kubernetes.io/projected/edc2af65-1515-4742-89e1-b49814016b09-kube-api-access-6bxrl\") pod \"service-ca-operator-544d749c8c-j7f9n\" (UID: \"edc2af65-1515-4742-89e1-b49814016b09\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j7f9n" Oct 30 23:06:09.643512 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.643488 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg5n4\" (UniqueName: \"kubernetes.io/projected/582d0663-6eba-4e69-81b0-fb25380f4c47-kube-api-access-vg5n4\") pod \"ingress-canary-nzkrw\" (UID: \"582d0663-6eba-4e69-81b0-fb25380f4c47\") " pod="openshift-ingress-canary/ingress-canary-nzkrw" Oct 30 23:06:09.830670 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.830617 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-lthcb" Oct 30 23:06:09.831811 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.831785 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:09.835649 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.835634 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-mpbb4" Oct 30 23:06:09.918065 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.917936 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-szm9n"] Oct 30 23:06:09.923732 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.923151 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j7f9n" Oct 30 23:06:09.926896 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:06:09.926866 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67e30e6a_a537_4e2c_a484_f15281828fec.slice/crio-4374a26c15def0481cf21509e6531d6926bb66c03e0edd6e8e5b64fd66fc22e9 WatchSource:0}: Error finding container 4374a26c15def0481cf21509e6531d6926bb66c03e0edd6e8e5b64fd66fc22e9: Status 404 returned error can't find the container with id 4374a26c15def0481cf21509e6531d6926bb66c03e0edd6e8e5b64fd66fc22e9 Oct 30 23:06:09.938143 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.938121 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nzkrw" Oct 30 23:06:09.963600 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.963578 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kbrxt"] Oct 30 23:06:09.974449 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:06:09.974404 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode32c5dc1_43ac_4ca5_a391_947b68d10d51.slice/crio-487de86185e608fc2799e6118eb17391576d26d64e37d4d9ef3ae300dbe115d0 WatchSource:0}: Error finding container 487de86185e608fc2799e6118eb17391576d26d64e37d4d9ef3ae300dbe115d0: Status 404 returned error can't find the container with id 487de86185e608fc2799e6118eb17391576d26d64e37d4d9ef3ae300dbe115d0 Oct 30 23:06:09.989605 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.988937 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0cb2a164-a8b5-4b93-b656-304550f04179-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-r26cd\" (UID: \"0cb2a164-a8b5-4b93-b656-304550f04179\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r26cd" Oct 30 23:06:09.989605 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.989009 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h5vcm\" (UID: \"f644db3c-e78a-42a8-9af3-c61194a21839\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5vcm" Oct 30 23:06:09.989605 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.989069 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dh6nh\" (UID: \"813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dh6nh" Oct 30 23:06:09.989605 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.989107 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f6c1ad7-10c4-4995-ae18-315d38184360-service-ca-bundle\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:09.989605 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:09.989112 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 23:06:09.989605 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.989139 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-tls\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:09.989605 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:09.989185 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cb2a164-a8b5-4b93-b656-304550f04179-networking-console-plugin-cert podName:0cb2a164-a8b5-4b93-b656-304550f04179 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:10.989165672 +0000 UTC m=+33.224590458 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0cb2a164-a8b5-4b93-b656-304550f04179-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-r26cd" (UID: "0cb2a164-a8b5-4b93-b656-304550f04179") : secret "networking-console-plugin-cert" not found Oct 30 23:06:09.989605 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.989214 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-metrics-certs\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:09.989605 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:09.989249 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:06:09.989605 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:09.989264 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9c458c7df-njhmv: secret "image-registry-tls" not found Oct 30 23:06:09.989605 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:09.989284 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-tls\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:09.989605 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:09.989314 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-tls podName:3ec44853-19be-4bbb-9840-bb4c96fd2a29 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:10.989297918 +0000 UTC m=+33.224722703 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-tls") pod "image-registry-9c458c7df-njhmv" (UID: "3ec44853-19be-4bbb-9840-bb4c96fd2a29") : secret "image-registry-tls" not found Oct 30 23:06:09.989605 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:09.989358 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:06:09.989605 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:09.989371 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f84d4ffd8-wx9nr: secret "image-registry-tls" not found Oct 30 23:06:09.989605 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:09.989408 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-tls podName:8cbb1813-35e1-49cb-8718-d0ba47d2350d nodeName:}" failed. No retries permitted until 2026-10-30 23:06:10.989395577 +0000 UTC m=+33.224820368 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-tls") pod "image-registry-6f84d4ffd8-wx9nr" (UID: "8cbb1813-35e1-49cb-8718-d0ba47d2350d") : secret "image-registry-tls" not found Oct 30 23:06:09.989605 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:09.989434 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 23:06:09.989605 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:09.989475 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 23:06:09.990499 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:09.989481 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-tls podName:f644db3c-e78a-42a8-9af3-c61194a21839 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:10.989466552 +0000 UTC m=+33.224891397 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h5vcm" (UID: "f644db3c-e78a-42a8-9af3-c61194a21839") : secret "samples-operator-tls" not found Oct 30 23:06:09.990499 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:09.989510 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-metrics-certs podName:5f6c1ad7-10c4-4995-ae18-315d38184360 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:10.989498701 +0000 UTC m=+33.224923505 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-metrics-certs") pod "router-default-5dd6f58df-tctqk" (UID: "5f6c1ad7-10c4-4995-ae18-315d38184360") : secret "router-metrics-certs-default" not found Oct 30 23:06:09.990499 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:09.989547 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 23:06:09.990499 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:09.989583 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f6c1ad7-10c4-4995-ae18-315d38184360-service-ca-bundle podName:5f6c1ad7-10c4-4995-ae18-315d38184360 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:10.989572992 +0000 UTC m=+33.224997785 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5f6c1ad7-10c4-4995-ae18-315d38184360-service-ca-bundle") pod "router-default-5dd6f58df-tctqk" (UID: "5f6c1ad7-10c4-4995-ae18-315d38184360") : configmap references non-existent config key: service-ca.crt Oct 30 23:06:09.990499 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:09.989600 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-cluster-monitoring-operator-tls podName:813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:10.989592063 +0000 UTC m=+33.225016839 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-dh6nh" (UID: "813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5") : secret "cluster-monitoring-operator-tls" not found Oct 30 23:06:10.010715 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:10.010466 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-kztwm"] Oct 30 23:06:10.016021 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:06:10.015994 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9a89b14_a8e8_45f2_84bd_dba21617499c.slice/crio-8cc1b5daf74f5223ffb2285e88662648c93d1d807a8cdc3775c9a4b57ec74333 WatchSource:0}: Error finding container 8cc1b5daf74f5223ffb2285e88662648c93d1d807a8cdc3775c9a4b57ec74333: Status 404 returned error can't find the container with id 8cc1b5daf74f5223ffb2285e88662648c93d1d807a8cdc3775c9a4b57ec74333 Oct 30 23:06:10.042254 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:10.042170 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-62rf9"] Oct 30 23:06:10.073003 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:10.072864 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chxn6"] Oct 30 23:06:10.077499 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:10.077432 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-g97cr"] Oct 30 23:06:10.082242 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:06:10.082166 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef598d5f_a066_4b2d_96bd_fbfc47741dbb.slice/crio-c249bb339a7fff2e18422f1f388959232c4d30a100cdd65db5dfd96c0da2140b WatchSource:0}: Error finding container c249bb339a7fff2e18422f1f388959232c4d30a100cdd65db5dfd96c0da2140b: Status 404 returned error can't find the container with id c249bb339a7fff2e18422f1f388959232c4d30a100cdd65db5dfd96c0da2140b Oct 30 23:06:10.090398 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:10.090381 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58d18552-3df9-47f4-9291-b0e82cd954d2-metrics-tls\") pod \"dns-default-j9spz\" (UID: \"58d18552-3df9-47f4-9291-b0e82cd954d2\") " pod="openshift-dns/dns-default-j9spz" Oct 30 23:06:10.090546 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:10.090529 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 23:06:10.090593 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:10.090585 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58d18552-3df9-47f4-9291-b0e82cd954d2-metrics-tls podName:58d18552-3df9-47f4-9291-b0e82cd954d2 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:11.090571713 +0000 UTC m=+33.325996481 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/58d18552-3df9-47f4-9291-b0e82cd954d2-metrics-tls") pod "dns-default-j9spz" (UID: "58d18552-3df9-47f4-9291-b0e82cd954d2") : secret "dns-default-metrics-tls" not found Oct 30 23:06:10.096220 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:10.096202 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:10.221639 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:10.221541 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-lthcb"] Oct 30 23:06:10.223912 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:06:10.223886 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a7f82b8_115d_43ed_a7a2_cb869a7e48e0.slice/crio-9e5aeafe6cf21dbb40184cceedf15ae29eb2517a993918361607d6e3f5dde5fb WatchSource:0}: Error finding container 9e5aeafe6cf21dbb40184cceedf15ae29eb2517a993918361607d6e3f5dde5fb: Status 404 returned error can't find the container with id 9e5aeafe6cf21dbb40184cceedf15ae29eb2517a993918361607d6e3f5dde5fb Oct 30 23:06:10.248355 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:10.248329 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mpbb4"] Oct 30 23:06:10.327973 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:10.327949 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-j7f9n"] Oct 30 23:06:10.332284 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:06:10.332231 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc2af65_1515_4742_89e1_b49814016b09.slice/crio-c835c85a86c44cbbcd65500910f97f39ea2a458b5919bacf36b6eb4485c9ef4b WatchSource:0}: Error finding container c835c85a86c44cbbcd65500910f97f39ea2a458b5919bacf36b6eb4485c9ef4b: Status 404 returned error can't find the container with id c835c85a86c44cbbcd65500910f97f39ea2a458b5919bacf36b6eb4485c9ef4b Oct 30 23:06:10.394323 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:10.394291 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-szm9n" event={"ID":"67e30e6a-a537-4e2c-a484-f15281828fec","Type":"ContainerStarted","Data":"e1198d141d2f9b2b0e58d3eb52b46b3ab81e400887bbc70dd5d4e02c028df72b"} Oct 30 23:06:10.394419 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:10.394350 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-szm9n" event={"ID":"67e30e6a-a537-4e2c-a484-f15281828fec","Type":"ContainerStarted","Data":"4374a26c15def0481cf21509e6531d6926bb66c03e0edd6e8e5b64fd66fc22e9"} Oct 30 23:06:10.395489 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:10.395454 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j7f9n" event={"ID":"edc2af65-1515-4742-89e1-b49814016b09","Type":"ContainerStarted","Data":"c835c85a86c44cbbcd65500910f97f39ea2a458b5919bacf36b6eb4485c9ef4b"} Oct 30 23:06:10.396593 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:10.396562 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-62rf9" event={"ID":"ad111ef6-4d18-4438-a0cc-e31a46804710","Type":"ContainerStarted","Data":"a959ec9ff3ac805313fedb7b37a657941adadffaec8ffc81bace9e5ebc780d02"} Oct 30 23:06:10.397882 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:10.397859 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-mpbb4" event={"ID":"624d7fe2-4960-48ba-b711-3d85e4bf0687","Type":"ContainerStarted","Data":"9345d13a29be91212f44438c97f157332624d25fb32ad5e37b4c6daab5890211"} Oct 30 23:06:10.397959 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:10.397895 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-mpbb4" event={"ID":"624d7fe2-4960-48ba-b711-3d85e4bf0687","Type":"ContainerStarted","Data":"0e986cf43acdc91e04b55a1643afd28a907823e006cc7c87146f051e1aa17524"} Oct 30 23:06:10.399105 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:10.399086 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-g97cr" event={"ID":"ef598d5f-a066-4b2d-96bd-fbfc47741dbb","Type":"ContainerStarted","Data":"c249bb339a7fff2e18422f1f388959232c4d30a100cdd65db5dfd96c0da2140b"} Oct 30 23:06:10.400138 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:10.400116 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chxn6" event={"ID":"6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893","Type":"ContainerStarted","Data":"b71dbcb555e71fae6aaffc5d8c19a7adc8dead3a2e5a786976813e1af918144a"} Oct 30 23:06:10.401459 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:10.401430 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-lthcb" event={"ID":"8a7f82b8-115d-43ed-a7a2-cb869a7e48e0","Type":"ContainerStarted","Data":"83a8a05d170664d2dc302063d87e89c0f6fb666d47c7d7c2189982a8aaaa647b"} Oct 30 23:06:10.401551 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:10.401473 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-lthcb" event={"ID":"8a7f82b8-115d-43ed-a7a2-cb869a7e48e0","Type":"ContainerStarted","Data":"9e5aeafe6cf21dbb40184cceedf15ae29eb2517a993918361607d6e3f5dde5fb"} Oct 30 23:06:10.402558 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:10.402537 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" event={"ID":"e9a89b14-a8e8-45f2-84bd-dba21617499c","Type":"ContainerStarted","Data":"8cc1b5daf74f5223ffb2285e88662648c93d1d807a8cdc3775c9a4b57ec74333"} Oct 30 23:06:10.403631 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:10.403612 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kbrxt" event={"ID":"e32c5dc1-43ac-4ca5-a391-947b68d10d51","Type":"ContainerStarted","Data":"487de86185e608fc2799e6118eb17391576d26d64e37d4d9ef3ae300dbe115d0"} Oct 30 23:06:10.474005 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:10.473990 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nzkrw"] Oct 30 23:06:10.476716 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:06:10.476689 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod582d0663_6eba_4e69_81b0_fb25380f4c47.slice/crio-8fe7a5b2e0ae165f7854fb568f1b3b75bc57b21564b5d529b3a2076811511adc WatchSource:0}: Error finding container 8fe7a5b2e0ae165f7854fb568f1b3b75bc57b21564b5d529b3a2076811511adc: Status 404 returned error can't find the container with id 8fe7a5b2e0ae165f7854fb568f1b3b75bc57b21564b5d529b3a2076811511adc Oct 30 23:06:10.997690 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:10.997657 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0cb2a164-a8b5-4b93-b656-304550f04179-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-r26cd\" (UID: \"0cb2a164-a8b5-4b93-b656-304550f04179\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r26cd" Oct 30 23:06:10.997850 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:10.997721 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h5vcm\" (UID: \"f644db3c-e78a-42a8-9af3-c61194a21839\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5vcm" Oct 30 23:06:10.997850 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:10.997758 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dh6nh\" (UID: \"813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dh6nh" Oct 30 23:06:10.997850 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:10.997792 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f6c1ad7-10c4-4995-ae18-315d38184360-service-ca-bundle\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:10.997850 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:10.997792 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 23:06:10.998079 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:10.997886 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 23:06:10.998079 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:10.997900 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cb2a164-a8b5-4b93-b656-304550f04179-networking-console-plugin-cert podName:0cb2a164-a8b5-4b93-b656-304550f04179 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:12.997851089 +0000 UTC m=+35.233275875 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0cb2a164-a8b5-4b93-b656-304550f04179-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-r26cd" (UID: "0cb2a164-a8b5-4b93-b656-304550f04179") : secret "networking-console-plugin-cert" not found Oct 30 23:06:10.998079 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:10.997894 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 23:06:10.998079 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:10.997943 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-tls\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:10.998079 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:10.997973 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-cluster-monitoring-operator-tls podName:813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:12.997956848 +0000 UTC m=+35.233381640 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-dh6nh" (UID: "813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5") : secret "cluster-monitoring-operator-tls" not found Oct 30 23:06:10.998079 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:10.998019 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f6c1ad7-10c4-4995-ae18-315d38184360-service-ca-bundle podName:5f6c1ad7-10c4-4995-ae18-315d38184360 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:12.998002657 +0000 UTC m=+35.233427433 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5f6c1ad7-10c4-4995-ae18-315d38184360-service-ca-bundle") pod "router-default-5dd6f58df-tctqk" (UID: "5f6c1ad7-10c4-4995-ae18-315d38184360") : configmap references non-existent config key: service-ca.crt Oct 30 23:06:10.998079 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:10.998018 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:06:10.998079 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:10.998063 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-tls podName:f644db3c-e78a-42a8-9af3-c61194a21839 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:12.99803235 +0000 UTC m=+35.233457121 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h5vcm" (UID: "f644db3c-e78a-42a8-9af3-c61194a21839") : secret "samples-operator-tls" not found Oct 30 23:06:10.998429 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:10.998058 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9c458c7df-njhmv: secret "image-registry-tls" not found Oct 30 23:06:10.998429 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:10.998118 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-metrics-certs\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:10.998429 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:10.998140 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-tls podName:3ec44853-19be-4bbb-9840-bb4c96fd2a29 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:12.99812308 +0000 UTC m=+35.233547851 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-tls") pod "image-registry-9c458c7df-njhmv" (UID: "3ec44853-19be-4bbb-9840-bb4c96fd2a29") : secret "image-registry-tls" not found Oct 30 23:06:10.998429 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:10.998181 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 23:06:10.998429 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:10.998219 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-tls\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:10.998429 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:10.998227 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-metrics-certs podName:5f6c1ad7-10c4-4995-ae18-315d38184360 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:12.99821569 +0000 UTC m=+35.233640471 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-metrics-certs") pod "router-default-5dd6f58df-tctqk" (UID: "5f6c1ad7-10c4-4995-ae18-315d38184360") : secret "router-metrics-certs-default" not found Oct 30 23:06:10.998429 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:10.998289 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:06:10.998429 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:10.998300 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f84d4ffd8-wx9nr: secret "image-registry-tls" not found Oct 30 23:06:10.998429 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:10.998336 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-tls podName:8cbb1813-35e1-49cb-8718-d0ba47d2350d nodeName:}" failed. No retries permitted until 2026-10-30 23:06:12.998324919 +0000 UTC m=+35.233749689 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-tls") pod "image-registry-6f84d4ffd8-wx9nr" (UID: "8cbb1813-35e1-49cb-8718-d0ba47d2350d") : secret "image-registry-tls" not found Oct 30 23:06:11.099386 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:11.099352 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58d18552-3df9-47f4-9291-b0e82cd954d2-metrics-tls\") pod \"dns-default-j9spz\" (UID: \"58d18552-3df9-47f4-9291-b0e82cd954d2\") " pod="openshift-dns/dns-default-j9spz" Oct 30 23:06:11.099806 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:11.099586 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 23:06:11.099806 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:11.099639 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58d18552-3df9-47f4-9291-b0e82cd954d2-metrics-tls podName:58d18552-3df9-47f4-9291-b0e82cd954d2 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:13.099623372 +0000 UTC m=+35.335048153 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/58d18552-3df9-47f4-9291-b0e82cd954d2-metrics-tls") pod "dns-default-j9spz" (UID: "58d18552-3df9-47f4-9291-b0e82cd954d2") : secret "dns-default-metrics-tls" not found Oct 30 23:06:11.409176 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:11.408559 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nzkrw" event={"ID":"582d0663-6eba-4e69-81b0-fb25380f4c47","Type":"ContainerStarted","Data":"8fe7a5b2e0ae165f7854fb568f1b3b75bc57b21564b5d529b3a2076811511adc"} Oct 30 23:06:11.461156 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:11.461098 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-mpbb4" podStartSLOduration=345.461080361 podStartE2EDuration="5m45.461080361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:06:11.346293512 +0000 UTC m=+33.581718304" watchObservedRunningTime="2026-10-30 23:06:11.461080361 +0000 UTC m=+33.696505155" Oct 30 23:06:11.461523 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:11.461485 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-szm9n" podStartSLOduration=345.461471668 podStartE2EDuration="5m45.461471668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:06:11.460752892 +0000 UTC m=+33.696177683" watchObservedRunningTime="2026-10-30 23:06:11.461471668 +0000 UTC m=+33.696896460" Oct 30 23:06:12.009090 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:12.008530 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/322e3a9b-911f-4cd8-8f43-71d56152e917-original-pull-secret\") pod \"global-pull-secret-syncer-7vrpr\" (UID: \"322e3a9b-911f-4cd8-8f43-71d56152e917\") " pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:06:12.009090 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:12.008590 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs\") pod \"network-metrics-daemon-htqtp\" (UID: \"dec2c5fa-4b9a-47ed-b005-04874761f419\") " pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:06:12.009090 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:12.008754 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 23:06:12.009090 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:12.008806 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs podName:dec2c5fa-4b9a-47ed-b005-04874761f419 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:44.008790481 +0000 UTC m=+66.244215253 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs") pod "network-metrics-daemon-htqtp" (UID: "dec2c5fa-4b9a-47ed-b005-04874761f419") : secret "metrics-daemon-secret" not found Oct 30 23:06:12.019245 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:12.019213 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/322e3a9b-911f-4cd8-8f43-71d56152e917-original-pull-secret\") pod \"global-pull-secret-syncer-7vrpr\" (UID: \"322e3a9b-911f-4cd8-8f43-71d56152e917\") " pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:06:12.110121 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:12.109548 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9nfl\" (UniqueName: \"kubernetes.io/projected/4f7d22b5-256e-47d8-947f-b89b47f14d08-kube-api-access-l9nfl\") pod \"network-check-target-xdlsv\" (UID: \"4f7d22b5-256e-47d8-947f-b89b47f14d08\") " pod="openshift-network-diagnostics/network-check-target-xdlsv" Oct 30 23:06:12.124408 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:12.124374 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9nfl\" (UniqueName: \"kubernetes.io/projected/4f7d22b5-256e-47d8-947f-b89b47f14d08-kube-api-access-l9nfl\") pod \"network-check-target-xdlsv\" (UID: \"4f7d22b5-256e-47d8-947f-b89b47f14d08\") " pod="openshift-network-diagnostics/network-check-target-xdlsv" Oct 30 23:06:12.198146 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:12.198112 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xdlsv" Oct 30 23:06:12.206150 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:12.206121 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:06:12.668803 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:12.668714 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-lthcb" podStartSLOduration=346.668694031 podStartE2EDuration="5m46.668694031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:06:11.576773223 +0000 UTC m=+33.812198013" watchObservedRunningTime="2026-10-30 23:06:12.668694031 +0000 UTC m=+34.904118823" Oct 30 23:06:12.670087 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:12.670017 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xdlsv"] Oct 30 23:06:12.718517 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:12.718390 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7vrpr"] Oct 30 23:06:13.020356 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:13.019273 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0cb2a164-a8b5-4b93-b656-304550f04179-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-r26cd\" (UID: \"0cb2a164-a8b5-4b93-b656-304550f04179\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r26cd" Oct 30 23:06:13.020356 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:13.019338 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h5vcm\" (UID: \"f644db3c-e78a-42a8-9af3-c61194a21839\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5vcm" Oct 30 23:06:13.020356 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:13.019373 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dh6nh\" (UID: \"813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dh6nh" Oct 30 23:06:13.020356 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:13.019410 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f6c1ad7-10c4-4995-ae18-315d38184360-service-ca-bundle\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:13.020356 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:13.019444 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-tls\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:13.020356 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:13.019477 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-metrics-certs\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:13.020356 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:13.019520 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-tls\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:13.020356 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:13.019704 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:06:13.020356 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:13.019721 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f84d4ffd8-wx9nr: secret "image-registry-tls" not found Oct 30 23:06:13.020356 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:13.019776 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-tls podName:8cbb1813-35e1-49cb-8718-d0ba47d2350d nodeName:}" failed. No retries permitted until 2026-10-30 23:06:17.019759396 +0000 UTC m=+39.255184168 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-tls") pod "image-registry-6f84d4ffd8-wx9nr" (UID: "8cbb1813-35e1-49cb-8718-d0ba47d2350d") : secret "image-registry-tls" not found Oct 30 23:06:13.020356 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:13.019861 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 23:06:13.020356 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:13.019898 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cb2a164-a8b5-4b93-b656-304550f04179-networking-console-plugin-cert podName:0cb2a164-a8b5-4b93-b656-304550f04179 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:17.019887721 +0000 UTC m=+39.255312503 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0cb2a164-a8b5-4b93-b656-304550f04179-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-r26cd" (UID: "0cb2a164-a8b5-4b93-b656-304550f04179") : secret "networking-console-plugin-cert" not found Oct 30 23:06:13.020356 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:13.019960 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 23:06:13.020356 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:13.019992 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-tls podName:f644db3c-e78a-42a8-9af3-c61194a21839 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:17.019981487 +0000 UTC m=+39.255406269 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h5vcm" (UID: "f644db3c-e78a-42a8-9af3-c61194a21839") : secret "samples-operator-tls" not found Oct 30 23:06:13.020356 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:13.020065 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 23:06:13.021123 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:13.020096 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-cluster-monitoring-operator-tls podName:813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:17.020086381 +0000 UTC m=+39.255511162 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-dh6nh" (UID: "813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5") : secret "cluster-monitoring-operator-tls" not found Oct 30 23:06:13.021123 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:13.020161 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f6c1ad7-10c4-4995-ae18-315d38184360-service-ca-bundle podName:5f6c1ad7-10c4-4995-ae18-315d38184360 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:17.020153433 +0000 UTC m=+39.255578214 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5f6c1ad7-10c4-4995-ae18-315d38184360-service-ca-bundle") pod "router-default-5dd6f58df-tctqk" (UID: "5f6c1ad7-10c4-4995-ae18-315d38184360") : configmap references non-existent config key: service-ca.crt Oct 30 23:06:13.021123 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:13.020207 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:06:13.021123 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:13.020217 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9c458c7df-njhmv: secret "image-registry-tls" not found Oct 30 23:06:13.021123 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:13.020238 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-tls podName:3ec44853-19be-4bbb-9840-bb4c96fd2a29 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:17.02023195 +0000 UTC m=+39.255656730 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-tls") pod "image-registry-9c458c7df-njhmv" (UID: "3ec44853-19be-4bbb-9840-bb4c96fd2a29") : secret "image-registry-tls" not found Oct 30 23:06:13.021123 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:13.020296 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 23:06:13.021123 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:13.020324 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-metrics-certs podName:5f6c1ad7-10c4-4995-ae18-315d38184360 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:17.020315715 +0000 UTC m=+39.255740498 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-metrics-certs") pod "router-default-5dd6f58df-tctqk" (UID: "5f6c1ad7-10c4-4995-ae18-315d38184360") : secret "router-metrics-certs-default" not found Oct 30 23:06:13.120938 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:13.120864 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58d18552-3df9-47f4-9291-b0e82cd954d2-metrics-tls\") pod \"dns-default-j9spz\" (UID: \"58d18552-3df9-47f4-9291-b0e82cd954d2\") " pod="openshift-dns/dns-default-j9spz" Oct 30 23:06:13.121329 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:13.121065 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 23:06:13.121329 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:13.121126 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58d18552-3df9-47f4-9291-b0e82cd954d2-metrics-tls podName:58d18552-3df9-47f4-9291-b0e82cd954d2 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:17.121106327 +0000 UTC m=+39.356531117 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/58d18552-3df9-47f4-9291-b0e82cd954d2-metrics-tls") pod "dns-default-j9spz" (UID: "58d18552-3df9-47f4-9291-b0e82cd954d2") : secret "dns-default-metrics-tls" not found Oct 30 23:06:13.196872 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:13.196836 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:13.413881 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:13.413761 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:13.426160 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:13.426114 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kbrxt" event={"ID":"e32c5dc1-43ac-4ca5-a391-947b68d10d51","Type":"ContainerStarted","Data":"cb1e28b6ad7606aecca25fd247b473b2e96ee1f7a34ce87834744f67b09088b0"} Oct 30 23:06:13.431147 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:13.431118 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7vrpr" event={"ID":"322e3a9b-911f-4cd8-8f43-71d56152e917","Type":"ContainerStarted","Data":"9e0f56088f54a7f4479099ef4bd5f80a2b25ff5657f33c8920d01d9857369059"} Oct 30 23:06:13.434823 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:13.434766 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xdlsv" event={"ID":"4f7d22b5-256e-47d8-947f-b89b47f14d08","Type":"ContainerStarted","Data":"622a3003f71c337888542d7ca03cfeea3d5ba41630ac893ca1b000a06ddd87b9"} Oct 30 23:06:13.542469 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:13.542406 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kbrxt" podStartSLOduration=317.69303488 podStartE2EDuration="5m20.542389091s" totalImagesPullingTime="2.849354211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:06:13.540833821 +0000 UTC m=+35.776258614" watchObservedRunningTime="2026-10-30 23:06:13.542389091 +0000 UTC m=+35.777813881" Oct 30 23:06:13.734802 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:13.734715 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:13.939491 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:13.939454 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:15.444448 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:15.444355 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/0.log" Oct 30 23:06:15.444448 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:15.444404 2570 generic.go:356] "Generic (PLEG): container finished" podID="e9a89b14-a8e8-45f2-84bd-dba21617499c" containerID="46a66c7b58a901622f56ccc8cf3da30d711dc61a403f55663f47a3727e6fca24" exitCode=255 Oct 30 23:06:15.444904 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:15.444518 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" event={"ID":"e9a89b14-a8e8-45f2-84bd-dba21617499c","Type":"ContainerDied","Data":"46a66c7b58a901622f56ccc8cf3da30d711dc61a403f55663f47a3727e6fca24"} Oct 30 23:06:15.444904 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:15.444730 2570 scope.go:122] "RemoveContainer" containerID="46a66c7b58a901622f56ccc8cf3da30d711dc61a403f55663f47a3727e6fca24" Oct 30 23:06:15.446988 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:15.446964 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-g97cr" event={"ID":"ef598d5f-a066-4b2d-96bd-fbfc47741dbb","Type":"ContainerStarted","Data":"18edf6ff4c8ba28a7b3c0865d398080e669631865a0de77558bb5c5c4457426b"} Oct 30 23:06:16.460359 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:16.460271 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chxn6" event={"ID":"6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893","Type":"ContainerStarted","Data":"1da7179d70e8597de30ed974576439dd55d684f2d10d3f4e1b4be85ce0ef0a37"} Oct 30 23:06:16.465096 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:16.465069 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/1.log" Oct 30 23:06:16.465757 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:16.465572 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/0.log" Oct 30 23:06:16.465757 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:16.465603 2570 generic.go:356] "Generic (PLEG): container finished" podID="e9a89b14-a8e8-45f2-84bd-dba21617499c" containerID="066dcc6dcd6ce40d14f2a0ba4f1a61ca739e7a5dbee2bfa6413b3068b117427d" exitCode=255 Oct 30 23:06:16.465757 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:16.465696 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" event={"ID":"e9a89b14-a8e8-45f2-84bd-dba21617499c","Type":"ContainerDied","Data":"066dcc6dcd6ce40d14f2a0ba4f1a61ca739e7a5dbee2bfa6413b3068b117427d"} Oct 30 23:06:16.465757 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:16.465735 2570 scope.go:122] "RemoveContainer" containerID="46a66c7b58a901622f56ccc8cf3da30d711dc61a403f55663f47a3727e6fca24" Oct 30 23:06:16.466103 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:16.466087 2570 scope.go:122] "RemoveContainer" containerID="066dcc6dcd6ce40d14f2a0ba4f1a61ca739e7a5dbee2bfa6413b3068b117427d" Oct 30 23:06:16.466308 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:16.466287 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-kztwm_openshift-console-operator(e9a89b14-a8e8-45f2-84bd-dba21617499c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" podUID="e9a89b14-a8e8-45f2-84bd-dba21617499c" Oct 30 23:06:16.472795 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:16.472770 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j7f9n" event={"ID":"edc2af65-1515-4742-89e1-b49814016b09","Type":"ContainerStarted","Data":"14104234ddbdd88cfcb38a94ecf6cfe9de0320fca2ed0f5b273eda0f508d680c"} Oct 30 23:06:16.578173 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:16.577667 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-g97cr" podStartSLOduration=316.521825886 podStartE2EDuration="5m21.577646988s" totalImagesPullingTime="5.055821102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:06:15.678025511 +0000 UTC m=+37.913450302" watchObservedRunningTime="2026-10-30 23:06:16.577646988 +0000 UTC m=+38.813071782" Oct 30 23:06:16.697774 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:16.697717 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j7f9n" podStartSLOduration=317.911044964 podStartE2EDuration="5m23.697703123s" totalImagesPullingTime="5.786658159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:06:16.696738259 +0000 UTC m=+38.932163050" watchObservedRunningTime="2026-10-30 23:06:16.697703123 +0000 UTC m=+38.933127914" Oct 30 23:06:16.698077 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:16.698017 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chxn6" podStartSLOduration=318.126121717 podStartE2EDuration="5m23.698002454s" totalImagesPullingTime="5.571880737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:06:16.577358389 +0000 UTC m=+38.812783180" watchObservedRunningTime="2026-10-30 23:06:16.698002454 +0000 UTC m=+38.933427267" Oct 30 23:06:16.943788 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:16.942402 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lfbzq"] Oct 30 23:06:16.967534 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:16.966799 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:17.005194 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.005103 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lfbzq"] Oct 30 23:06:17.020714 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.020582 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 23:06:17.020714 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.020686 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 23:06:17.021072 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.020805 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-v482q\"" Oct 30 23:06:17.060850 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.060822 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0cb2a164-a8b5-4b93-b656-304550f04179-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-r26cd\" (UID: \"0cb2a164-a8b5-4b93-b656-304550f04179\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r26cd" Oct 30 23:06:17.061007 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.060861 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/601f3571-aa13-43c6-8b5a-393029720fc3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:17.061007 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.060878 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/601f3571-aa13-43c6-8b5a-393029720fc3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:17.061007 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.060914 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h5vcm\" (UID: \"f644db3c-e78a-42a8-9af3-c61194a21839\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5vcm" Oct 30 23:06:17.061007 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.060936 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dh6nh\" (UID: \"813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dh6nh" Oct 30 23:06:17.061007 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.060961 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f6c1ad7-10c4-4995-ae18-315d38184360-service-ca-bundle\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:17.061007 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.060978 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-tls\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:17.061007 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.060995 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/601f3571-aa13-43c6-8b5a-393029720fc3-data-volume\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:17.061007 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:17.060998 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 23:06:17.061007 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.061011 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdsrp\" (UniqueName: \"kubernetes.io/projected/601f3571-aa13-43c6-8b5a-393029720fc3-kube-api-access-rdsrp\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:17.061366 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.061056 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-metrics-certs\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:17.061366 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.061088 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-tls\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:17.061366 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.061125 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/601f3571-aa13-43c6-8b5a-393029720fc3-crio-socket\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:17.061366 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:17.061246 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 23:06:17.061366 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:17.061285 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-tls podName:f644db3c-e78a-42a8-9af3-c61194a21839 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:25.061272925 +0000 UTC m=+47.296697695 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h5vcm" (UID: "f644db3c-e78a-42a8-9af3-c61194a21839") : secret "samples-operator-tls" not found Oct 30 23:06:17.061366 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:17.061319 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 23:06:17.061366 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:17.061340 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-cluster-monitoring-operator-tls podName:813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:25.061334478 +0000 UTC m=+47.296759247 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-dh6nh" (UID: "813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5") : secret "cluster-monitoring-operator-tls" not found Oct 30 23:06:17.061611 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:17.061386 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 23:06:17.061611 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:17.061412 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f6c1ad7-10c4-4995-ae18-315d38184360-service-ca-bundle podName:5f6c1ad7-10c4-4995-ae18-315d38184360 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:25.06139635 +0000 UTC m=+47.296821117 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5f6c1ad7-10c4-4995-ae18-315d38184360-service-ca-bundle") pod "router-default-5dd6f58df-tctqk" (UID: "5f6c1ad7-10c4-4995-ae18-315d38184360") : configmap references non-existent config key: service-ca.crt Oct 30 23:06:17.061611 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:17.061427 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-metrics-certs podName:5f6c1ad7-10c4-4995-ae18-315d38184360 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:25.061420799 +0000 UTC m=+47.296845567 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-metrics-certs") pod "router-default-5dd6f58df-tctqk" (UID: "5f6c1ad7-10c4-4995-ae18-315d38184360") : secret "router-metrics-certs-default" not found Oct 30 23:06:17.061611 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:17.061435 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:06:17.061611 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:17.061443 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f84d4ffd8-wx9nr: secret "image-registry-tls" not found Oct 30 23:06:17.061611 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:17.061460 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-tls podName:8cbb1813-35e1-49cb-8718-d0ba47d2350d nodeName:}" failed. No retries permitted until 2026-10-30 23:06:25.061454149 +0000 UTC m=+47.296878916 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-tls") pod "image-registry-6f84d4ffd8-wx9nr" (UID: "8cbb1813-35e1-49cb-8718-d0ba47d2350d") : secret "image-registry-tls" not found Oct 30 23:06:17.061611 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:17.061470 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cb2a164-a8b5-4b93-b656-304550f04179-networking-console-plugin-cert podName:0cb2a164-a8b5-4b93-b656-304550f04179 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:25.061464064 +0000 UTC m=+47.296888832 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0cb2a164-a8b5-4b93-b656-304550f04179-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-r26cd" (UID: "0cb2a164-a8b5-4b93-b656-304550f04179") : secret "networking-console-plugin-cert" not found Oct 30 23:06:17.061611 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:17.061481 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:06:17.061611 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:17.061491 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9c458c7df-njhmv: secret "image-registry-tls" not found Oct 30 23:06:17.061611 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:17.061513 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-tls podName:3ec44853-19be-4bbb-9840-bb4c96fd2a29 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:25.061506649 +0000 UTC m=+47.296931417 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-tls") pod "image-registry-9c458c7df-njhmv" (UID: "3ec44853-19be-4bbb-9840-bb4c96fd2a29") : secret "image-registry-tls" not found Oct 30 23:06:17.163013 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.162142 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/601f3571-aa13-43c6-8b5a-393029720fc3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:17.163013 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.162175 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/601f3571-aa13-43c6-8b5a-393029720fc3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:17.163013 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.162238 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/601f3571-aa13-43c6-8b5a-393029720fc3-data-volume\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:17.163013 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.162256 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdsrp\" (UniqueName: \"kubernetes.io/projected/601f3571-aa13-43c6-8b5a-393029720fc3-kube-api-access-rdsrp\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:17.163013 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.162368 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58d18552-3df9-47f4-9291-b0e82cd954d2-metrics-tls\") pod \"dns-default-j9spz\" (UID: \"58d18552-3df9-47f4-9291-b0e82cd954d2\") " pod="openshift-dns/dns-default-j9spz" Oct 30 23:06:17.163013 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.162389 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/601f3571-aa13-43c6-8b5a-393029720fc3-crio-socket\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:17.163013 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.162504 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/601f3571-aa13-43c6-8b5a-393029720fc3-crio-socket\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:17.163013 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:17.162578 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 23:06:17.163013 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:17.162649 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/601f3571-aa13-43c6-8b5a-393029720fc3-kube-rbac-proxy-cm podName:601f3571-aa13-43c6-8b5a-393029720fc3 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:17.662608383 +0000 UTC m=+39.898033164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/601f3571-aa13-43c6-8b5a-393029720fc3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lfbzq" (UID: "601f3571-aa13-43c6-8b5a-393029720fc3") : configmap "kube-rbac-proxy" not found Oct 30 23:06:17.163013 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:17.162725 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 23:06:17.163013 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:17.162754 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/601f3571-aa13-43c6-8b5a-393029720fc3-insights-runtime-extractor-tls podName:601f3571-aa13-43c6-8b5a-393029720fc3 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:17.662746234 +0000 UTC m=+39.898171015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/601f3571-aa13-43c6-8b5a-393029720fc3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lfbzq" (UID: "601f3571-aa13-43c6-8b5a-393029720fc3") : secret "insights-runtime-extractor-tls" not found Oct 30 23:06:17.164502 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:17.163239 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 23:06:17.164502 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:17.163308 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58d18552-3df9-47f4-9291-b0e82cd954d2-metrics-tls podName:58d18552-3df9-47f4-9291-b0e82cd954d2 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:25.163290584 +0000 UTC m=+47.398715376 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/58d18552-3df9-47f4-9291-b0e82cd954d2-metrics-tls") pod "dns-default-j9spz" (UID: "58d18552-3df9-47f4-9291-b0e82cd954d2") : secret "dns-default-metrics-tls" not found Oct 30 23:06:17.164743 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.164722 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/601f3571-aa13-43c6-8b5a-393029720fc3-data-volume\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:17.224024 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.223922 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdsrp\" (UniqueName: \"kubernetes.io/projected/601f3571-aa13-43c6-8b5a-393029720fc3-kube-api-access-rdsrp\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:17.270418 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.270383 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:17.478754 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.478689 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/1.log" Oct 30 23:06:17.479224 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.479121 2570 scope.go:122] "RemoveContainer" containerID="066dcc6dcd6ce40d14f2a0ba4f1a61ca739e7a5dbee2bfa6413b3068b117427d" Oct 30 23:06:17.479224 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.479178 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xlp5h"] Oct 30 23:06:17.479355 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:17.479330 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-kztwm_openshift-console-operator(e9a89b14-a8e8-45f2-84bd-dba21617499c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" podUID="e9a89b14-a8e8-45f2-84bd-dba21617499c" Oct 30 23:06:17.492777 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.492746 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nzkrw" event={"ID":"582d0663-6eba-4e69-81b0-fb25380f4c47","Type":"ContainerStarted","Data":"2ef4d5ddccafa157e8612ffabb2b4c20399cb18df4cec3c89d5f7585829d3511"} Oct 30 23:06:17.492918 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.492849 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xlp5h" Oct 30 23:06:17.509195 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.509013 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:17.539083 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.539029 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xlp5h"] Oct 30 23:06:17.565961 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.565926 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b6bz\" (UniqueName: \"kubernetes.io/projected/bcddee82-a2d2-471b-badb-0295456e6a90-kube-api-access-4b6bz\") pod \"migrator-66df5d496b-xlp5h\" (UID: \"bcddee82-a2d2-471b-badb-0295456e6a90\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xlp5h" Oct 30 23:06:17.589420 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.589396 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 23:06:17.589538 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.589450 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 23:06:17.644447 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.644408 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-w9j6t\"" Oct 30 23:06:17.667625 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.667585 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/601f3571-aa13-43c6-8b5a-393029720fc3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:17.667761 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.667634 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/601f3571-aa13-43c6-8b5a-393029720fc3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:17.667761 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.667748 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b6bz\" (UniqueName: \"kubernetes.io/projected/bcddee82-a2d2-471b-badb-0295456e6a90-kube-api-access-4b6bz\") pod \"migrator-66df5d496b-xlp5h\" (UID: \"bcddee82-a2d2-471b-badb-0295456e6a90\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xlp5h" Oct 30 23:06:17.668159 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:17.668142 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 23:06:17.668216 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:17.668207 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/601f3571-aa13-43c6-8b5a-393029720fc3-kube-rbac-proxy-cm podName:601f3571-aa13-43c6-8b5a-393029720fc3 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:18.668188141 +0000 UTC m=+40.903612912 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/601f3571-aa13-43c6-8b5a-393029720fc3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lfbzq" (UID: "601f3571-aa13-43c6-8b5a-393029720fc3") : configmap "kube-rbac-proxy" not found Oct 30 23:06:17.668295 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:17.668284 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 23:06:17.668330 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:17.668322 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/601f3571-aa13-43c6-8b5a-393029720fc3-insights-runtime-extractor-tls podName:601f3571-aa13-43c6-8b5a-393029720fc3 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:18.668310526 +0000 UTC m=+40.903735318 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/601f3571-aa13-43c6-8b5a-393029720fc3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lfbzq" (UID: "601f3571-aa13-43c6-8b5a-393029720fc3") : secret "insights-runtime-extractor-tls" not found Oct 30 23:06:17.713946 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.713885 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nzkrw" podStartSLOduration=1.81485486 podStartE2EDuration="8.713865031s" totalImagesPullingTime="6.899010171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:06:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:06:17.712871744 +0000 UTC m=+39.948296535" watchObservedRunningTime="2026-10-30 23:06:17.713865031 +0000 UTC m=+39.949289822" Oct 30 23:06:17.730605 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.730517 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b6bz\" (UniqueName: \"kubernetes.io/projected/bcddee82-a2d2-471b-badb-0295456e6a90-kube-api-access-4b6bz\") pod \"migrator-66df5d496b-xlp5h\" (UID: \"bcddee82-a2d2-471b-badb-0295456e6a90\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xlp5h" Oct 30 23:06:17.734912 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.734876 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:17.816204 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.816169 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xlp5h" Oct 30 23:06:17.939783 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:17.939753 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:18.172117 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:18.172018 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:18.202815 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:18.202791 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xlp5h"] Oct 30 23:06:18.205754 ip-10-0-130-177 kubenswrapper[2570]: W1030 23:06:18.205723 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcddee82_a2d2_471b_badb_0295456e6a90.slice/crio-724f369db6e1b26e89582a68d830fd9f8b170a494d63ea2253ef416049518117 WatchSource:0}: Error finding container 724f369db6e1b26e89582a68d830fd9f8b170a494d63ea2253ef416049518117: Status 404 returned error can't find the container with id 724f369db6e1b26e89582a68d830fd9f8b170a494d63ea2253ef416049518117 Oct 30 23:06:18.384933 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:18.384894 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:18.485701 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:18.485616 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xlp5h" event={"ID":"bcddee82-a2d2-471b-badb-0295456e6a90","Type":"ContainerStarted","Data":"724f369db6e1b26e89582a68d830fd9f8b170a494d63ea2253ef416049518117"} Oct 30 23:06:18.487397 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:18.487369 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-62rf9" event={"ID":"ad111ef6-4d18-4438-a0cc-e31a46804710","Type":"ContainerStarted","Data":"8d5a2b9fef8a1fd78c1e2f91ab6a914d7651f22486b911fc409cd62eeba53985"} Oct 30 23:06:18.603344 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:18.603288 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-62rf9" podStartSLOduration=314.103650349 podStartE2EDuration="5m21.603270437s" totalImagesPullingTime="7.499620088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:06:18.601856406 +0000 UTC m=+40.837281211" watchObservedRunningTime="2026-10-30 23:06:18.603270437 +0000 UTC m=+40.838695228" Oct 30 23:06:18.679016 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:18.678989 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/601f3571-aa13-43c6-8b5a-393029720fc3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:18.679170 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:18.679028 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/601f3571-aa13-43c6-8b5a-393029720fc3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:18.679170 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:18.679122 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 23:06:18.679170 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:18.679150 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 23:06:18.679304 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:18.679201 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/601f3571-aa13-43c6-8b5a-393029720fc3-kube-rbac-proxy-cm podName:601f3571-aa13-43c6-8b5a-393029720fc3 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:20.679182773 +0000 UTC m=+42.914607557 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/601f3571-aa13-43c6-8b5a-393029720fc3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lfbzq" (UID: "601f3571-aa13-43c6-8b5a-393029720fc3") : configmap "kube-rbac-proxy" not found Oct 30 23:06:18.679373 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:18.679359 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/601f3571-aa13-43c6-8b5a-393029720fc3-insights-runtime-extractor-tls podName:601f3571-aa13-43c6-8b5a-393029720fc3 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:20.679339331 +0000 UTC m=+42.914764102 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/601f3571-aa13-43c6-8b5a-393029720fc3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lfbzq" (UID: "601f3571-aa13-43c6-8b5a-393029720fc3") : secret "insights-runtime-extractor-tls" not found Oct 30 23:06:19.371804 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:19.371652 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:19.493122 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:19.493073 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xdlsv" event={"ID":"4f7d22b5-256e-47d8-947f-b89b47f14d08","Type":"ContainerStarted","Data":"0ab8637686f28a22a45ef7e73c4410fa3a0a9c1590f0b6bc6213355154e4ee0f"} Oct 30 23:06:19.563542 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:19.563510 2570 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" Oct 30 23:06:19.563687 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:19.563576 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" Oct 30 23:06:19.564406 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:19.564382 2570 scope.go:122] "RemoveContainer" containerID="066dcc6dcd6ce40d14f2a0ba4f1a61ca739e7a5dbee2bfa6413b3068b117427d" Oct 30 23:06:19.564787 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:19.564756 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-kztwm_openshift-console-operator(e9a89b14-a8e8-45f2-84bd-dba21617499c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" podUID="e9a89b14-a8e8-45f2-84bd-dba21617499c" Oct 30 23:06:19.611842 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:19.611783 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xdlsv" podStartSLOduration=34.802565138 podStartE2EDuration="40.611764043s" totalImagesPullingTime="5.809198905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:05:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:06:19.610118845 +0000 UTC m=+41.845543635" watchObservedRunningTime="2026-10-30 23:06:19.611764043 +0000 UTC m=+41.847188835" Oct 30 23:06:20.130447 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:20.130419 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:20.496852 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:20.496828 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xdlsv" Oct 30 23:06:20.698169 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:20.698092 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/601f3571-aa13-43c6-8b5a-393029720fc3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:20.698169 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:20.698122 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/601f3571-aa13-43c6-8b5a-393029720fc3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:20.698354 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:20.698216 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 23:06:20.698354 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:20.698228 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 23:06:20.698354 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:20.698271 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/601f3571-aa13-43c6-8b5a-393029720fc3-insights-runtime-extractor-tls podName:601f3571-aa13-43c6-8b5a-393029720fc3 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:24.698259886 +0000 UTC m=+46.933684667 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/601f3571-aa13-43c6-8b5a-393029720fc3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lfbzq" (UID: "601f3571-aa13-43c6-8b5a-393029720fc3") : secret "insights-runtime-extractor-tls" not found Oct 30 23:06:20.698354 ip-10-0-130-177 kubenswrapper[2570]: E1030 23:06:20.698284 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/601f3571-aa13-43c6-8b5a-393029720fc3-kube-rbac-proxy-cm podName:601f3571-aa13-43c6-8b5a-393029720fc3 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:24.698278425 +0000 UTC m=+46.933703193 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/601f3571-aa13-43c6-8b5a-393029720fc3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lfbzq" (UID: "601f3571-aa13-43c6-8b5a-393029720fc3") : configmap "kube-rbac-proxy" not found Oct 30 23:06:20.747014 ip-10-0-130-177 kubenswrapper[2570]: I1030 23:06:20.746100 2570 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 23:06:20.746317 ip-10-0-130-177 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 23:06:20.767415 ip-10-0-130-177 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 23:06:20.767586 ip-10-0-130-177 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 23:06:20.767637 ip-10-0-130-177 systemd[1]: kubelet.service: Consumed 3.113s CPU time. Oct 30 23:06:20.779677 ip-10-0-130-177 systemd[1]: Starting Kubernetes Kubelet... Oct 30 23:06:20.884163 ip-10-0-130-177 kubenswrapper[4952]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:06:20.884163 ip-10-0-130-177 kubenswrapper[4952]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 23:06:20.884163 ip-10-0-130-177 kubenswrapper[4952]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:06:20.884648 ip-10-0-130-177 kubenswrapper[4952]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:06:20.886873 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886794 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:06:20.886873 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886873 4952 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:06:20.886873 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886876 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:06:20.887004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886880 4952 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:06:20.887004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886883 4952 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:06:20.887004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886886 4952 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:06:20.887004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886889 4952 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:06:20.887004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886892 4952 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:06:20.887004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886894 4952 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:06:20.887004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886897 4952 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:06:20.887004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886899 4952 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:06:20.887004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886902 4952 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:06:20.887004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886905 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:06:20.887004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886908 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:06:20.887004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886911 4952 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:06:20.887004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886913 4952 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:06:20.887004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886916 4952 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:06:20.887004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886919 4952 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:06:20.887004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886921 4952 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:06:20.887004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886929 4952 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:06:20.887004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886932 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:06:20.887004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886935 4952 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:06:20.887004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886937 4952 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:06:20.887522 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886940 4952 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:06:20.887522 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886942 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:06:20.887522 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886945 4952 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:06:20.887522 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886948 4952 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:06:20.887522 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886950 4952 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:06:20.887522 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886953 4952 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:06:20.887522 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886956 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:06:20.887522 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886959 4952 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:06:20.887522 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886962 4952 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:06:20.887522 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886965 4952 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:06:20.887522 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886967 4952 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:06:20.887522 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886970 4952 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:06:20.887522 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886972 4952 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:06:20.887522 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886975 4952 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:06:20.887522 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886977 4952 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:06:20.887522 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886980 4952 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:06:20.887522 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886983 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:06:20.887522 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886985 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:06:20.887522 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886988 4952 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:06:20.887522 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886991 4952 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:06:20.888059 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886994 4952 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:06:20.888059 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886996 4952 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:06:20.888059 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.886999 4952 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:06:20.888059 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887002 4952 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:06:20.888059 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887004 4952 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:06:20.888059 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887007 4952 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:06:20.888059 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887010 4952 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:06:20.888059 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887014 4952 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:06:20.888059 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887017 4952 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:06:20.888059 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887020 4952 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:06:20.888059 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887022 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:06:20.888059 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887025 4952 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:06:20.888059 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887028 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:06:20.888059 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887030 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:06:20.888059 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887053 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:06:20.888059 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887057 4952 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:06:20.888059 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887061 4952 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:06:20.888059 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887066 4952 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:06:20.888059 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887069 4952 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:06:20.888555 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887072 4952 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:06:20.888555 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887075 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:06:20.888555 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887079 4952 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:06:20.888555 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887081 4952 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:06:20.888555 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887084 4952 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:06:20.888555 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887087 4952 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:06:20.888555 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887090 4952 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:06:20.888555 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887092 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:06:20.888555 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887095 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:06:20.888555 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887098 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:06:20.888555 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887101 4952 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:06:20.888555 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887103 4952 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:06:20.888555 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887106 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:06:20.888555 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887108 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:06:20.888555 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887111 4952 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:06:20.888555 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887113 4952 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:06:20.888555 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887116 4952 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:06:20.888555 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887119 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:06:20.888555 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887121 4952 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:06:20.889024 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887132 4952 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:06:20.889024 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887135 4952 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:06:20.889024 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887138 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:06:20.889024 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887141 4952 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:06:20.889024 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887143 4952 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:06:20.889024 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887146 4952 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:06:20.889024 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887149 4952 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:06:20.889024 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887152 4952 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:06:20.889024 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887154 4952 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:06:20.889024 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887157 4952 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:06:20.889024 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887160 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:06:20.889024 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887162 4952 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:06:20.889024 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887165 4952 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:06:20.889024 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887167 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:06:20.889024 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887170 4952 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:06:20.889024 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887173 4952 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:06:20.889024 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887177 4952 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:06:20.889024 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887181 4952 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:06:20.889024 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887185 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:06:20.889526 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887189 4952 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:06:20.889526 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887192 4952 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:06:20.889526 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887195 4952 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:06:20.889526 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887198 4952 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:06:20.889526 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887201 4952 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:06:20.889526 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887204 4952 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:06:20.889526 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887984 4952 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:06:20.889526 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887991 4952 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:06:20.889526 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887995 4952 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:06:20.889526 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.887997 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:06:20.889526 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888000 4952 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:06:20.889526 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888004 4952 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:06:20.889526 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888008 4952 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:06:20.889526 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888012 4952 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:06:20.889526 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888015 4952 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:06:20.889526 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888018 4952 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:06:20.889526 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888021 4952 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:06:20.889526 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888023 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:06:20.889526 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888026 4952 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:06:20.889994 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888028 4952 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:06:20.889994 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888031 4952 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:06:20.889994 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888049 4952 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:06:20.889994 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888052 4952 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:06:20.889994 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888055 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:06:20.889994 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888058 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:06:20.889994 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888061 4952 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:06:20.889994 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888064 4952 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:06:20.889994 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888066 4952 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:06:20.889994 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888069 4952 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:06:20.889994 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888072 4952 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:06:20.889994 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888075 4952 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:06:20.889994 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888078 4952 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:06:20.889994 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888080 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:06:20.889994 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888083 4952 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:06:20.889994 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888085 4952 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:06:20.889994 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888088 4952 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:06:20.889994 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888090 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:06:20.889994 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888093 4952 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:06:20.890484 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888096 4952 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:06:20.890484 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888099 4952 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:06:20.890484 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888101 4952 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:06:20.890484 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888104 4952 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:06:20.890484 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888106 4952 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:06:20.890484 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888109 4952 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:06:20.890484 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888111 4952 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:06:20.890484 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888114 4952 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:06:20.890484 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888116 4952 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:06:20.890484 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888120 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:06:20.890484 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888122 4952 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:06:20.890484 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888125 4952 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:06:20.890484 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888128 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:06:20.890484 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888130 4952 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:06:20.890484 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888133 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:06:20.890484 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888135 4952 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:06:20.890484 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888138 4952 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:06:20.890484 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888141 4952 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:06:20.890484 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888144 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:06:20.890484 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888146 4952 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:06:20.890996 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888149 4952 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:06:20.890996 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888151 4952 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:06:20.890996 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888154 4952 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:06:20.890996 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888157 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:06:20.890996 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888159 4952 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:06:20.890996 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888162 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:06:20.890996 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888165 4952 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:06:20.890996 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888169 4952 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:06:20.890996 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888172 4952 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:06:20.890996 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888175 4952 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:06:20.890996 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888178 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:06:20.890996 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888181 4952 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:06:20.890996 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888184 4952 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:06:20.890996 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888186 4952 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:06:20.890996 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888189 4952 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:06:20.890996 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888191 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:06:20.890996 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888195 4952 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:06:20.890996 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888197 4952 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:06:20.890996 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888200 4952 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:06:20.891505 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888202 4952 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:06:20.891505 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888205 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:06:20.891505 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888208 4952 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:06:20.891505 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888211 4952 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:06:20.891505 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888213 4952 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:06:20.891505 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888216 4952 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:06:20.891505 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888218 4952 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:06:20.891505 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888221 4952 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:06:20.891505 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888224 4952 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:06:20.891505 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888227 4952 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:06:20.891505 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888229 4952 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:06:20.891505 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888232 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:06:20.891505 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888235 4952 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:06:20.891505 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888237 4952 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:06:20.891505 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888240 4952 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:06:20.891505 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888243 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:06:20.891505 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888245 4952 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:06:20.891505 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888248 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:06:20.891505 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888251 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:06:20.891505 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888253 4952 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:06:20.892010 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888256 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:06:20.892010 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888258 4952 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:06:20.892010 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888261 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:06:20.892010 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888263 4952 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:06:20.892010 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888266 4952 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:06:20.892010 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888268 4952 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:06:20.892010 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888271 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:06:20.892010 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888274 4952 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:06:20.892010 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888276 4952 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:06:20.892010 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888279 4952 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:06:20.892010 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888281 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:06:20.892010 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888284 4952 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:06:20.892010 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888286 4952 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:06:20.892010 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888289 4952 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:06:20.892010 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888292 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:06:20.892010 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888837 4952 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:06:20.892010 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888844 4952 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:06:20.892010 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888848 4952 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:06:20.892010 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888851 4952 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:06:20.892517 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888854 4952 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:06:20.892517 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888857 4952 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:06:20.892517 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888860 4952 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:06:20.892517 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888863 4952 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:06:20.892517 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888865 4952 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:06:20.892517 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888868 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:06:20.892517 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888871 4952 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:06:20.892517 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888874 4952 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:06:20.892517 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888876 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:06:20.892517 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888879 4952 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:06:20.892517 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888882 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:06:20.892517 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888884 4952 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:06:20.892517 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888887 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:06:20.892517 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888892 4952 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:06:20.892517 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888895 4952 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:06:20.892517 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888898 4952 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:06:20.892517 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888901 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:06:20.892517 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888905 4952 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:06:20.892517 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888907 4952 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:06:20.893078 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888910 4952 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:06:20.893078 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888913 4952 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:06:20.893078 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888915 4952 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:06:20.893078 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888921 4952 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:06:20.893078 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888923 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:06:20.893078 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888926 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:06:20.893078 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888928 4952 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:06:20.893078 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888931 4952 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:06:20.893078 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888933 4952 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:06:20.893078 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888936 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:06:20.893078 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888939 4952 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:06:20.893078 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888942 4952 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:06:20.893078 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888944 4952 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:06:20.893078 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888947 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:06:20.893078 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888949 4952 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:06:20.893078 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888952 4952 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:06:20.893078 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888955 4952 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:06:20.893078 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888957 4952 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:06:20.893078 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888960 4952 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:06:20.893580 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888963 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:06:20.893580 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888966 4952 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:06:20.893580 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888968 4952 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:06:20.893580 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888971 4952 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:06:20.893580 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888974 4952 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:06:20.893580 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888976 4952 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:06:20.893580 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888979 4952 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:06:20.893580 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888982 4952 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:06:20.893580 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888984 4952 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:06:20.893580 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888987 4952 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:06:20.893580 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888990 4952 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:06:20.893580 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888992 4952 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:06:20.893580 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888995 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:06:20.893580 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.888997 4952 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:06:20.893580 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889000 4952 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:06:20.893580 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889003 4952 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:06:20.893580 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889007 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:06:20.893580 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889009 4952 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:06:20.893580 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889012 4952 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:06:20.893580 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889014 4952 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:06:20.894080 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889017 4952 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:06:20.894080 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889019 4952 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:06:20.894080 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889022 4952 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:06:20.894080 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889025 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:06:20.894080 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889028 4952 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:06:20.894080 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889031 4952 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:06:20.894080 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889054 4952 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:06:20.894080 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889057 4952 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:06:20.894080 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889060 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:06:20.894080 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889062 4952 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:06:20.894080 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889065 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:06:20.894080 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889068 4952 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:06:20.894080 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889070 4952 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:06:20.894080 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889073 4952 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:06:20.894080 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889075 4952 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:06:20.894080 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889078 4952 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:06:20.894080 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889082 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:06:20.894080 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889086 4952 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:06:20.894080 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889089 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:06:20.894080 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889092 4952 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:06:20.894577 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889094 4952 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:06:20.894577 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889097 4952 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:06:20.894577 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889099 4952 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:06:20.894577 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889102 4952 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:06:20.894577 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889106 4952 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:06:20.894577 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889109 4952 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:06:20.894577 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889112 4952 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:06:20.894577 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889115 4952 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:06:20.894577 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889119 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:06:20.894577 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889122 4952 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:06:20.894577 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889124 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:06:20.894577 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889127 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:06:20.894577 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889129 4952 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:06:20.894577 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889132 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:06:20.894577 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889134 4952 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:06:20.894577 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889137 4952 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:06:20.894577 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889140 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:06:20.894577 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889143 4952 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:06:20.894577 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889145 4952 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:06:20.894577 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889148 4952 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:06:20.895135 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889151 4952 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:06:20.895135 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889153 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:06:20.895135 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889156 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:06:20.895135 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889158 4952 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:06:20.895135 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889265 4952 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 23:06:20.895135 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889273 4952 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 23:06:20.895135 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889279 4952 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 23:06:20.895135 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889284 4952 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 23:06:20.895135 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889290 4952 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 23:06:20.895135 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889294 4952 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 23:06:20.895135 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889298 4952 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 23:06:20.895135 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889302 4952 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 23:06:20.895135 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889305 4952 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 23:06:20.895135 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889308 4952 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 23:06:20.895135 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889312 4952 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 23:06:20.895135 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889315 4952 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 23:06:20.895135 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889318 4952 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 23:06:20.895135 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889321 4952 flags.go:64] FLAG: --cgroup-root="" Oct 30 23:06:20.895135 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889324 4952 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 23:06:20.895135 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889327 4952 flags.go:64] FLAG: --client-ca-file="" Oct 30 23:06:20.895135 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889331 4952 flags.go:64] FLAG: --cloud-provider="external" Oct 30 23:06:20.895135 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889334 4952 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 23:06:20.895135 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889337 4952 flags.go:64] FLAG: --cluster-domain="" Oct 30 23:06:20.895687 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889340 4952 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 23:06:20.895687 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889343 4952 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 23:06:20.895687 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889346 4952 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 23:06:20.895687 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889350 4952 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 23:06:20.895687 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889354 4952 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 23:06:20.895687 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889356 4952 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 23:06:20.895687 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889360 4952 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 23:06:20.895687 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889364 4952 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 23:06:20.895687 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889367 4952 flags.go:64] FLAG: --contention-profiling="false" Oct 30 23:06:20.895687 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889370 4952 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 23:06:20.895687 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889373 4952 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 23:06:20.895687 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889376 4952 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 23:06:20.895687 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889379 4952 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 23:06:20.895687 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889383 4952 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 23:06:20.895687 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889386 4952 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 23:06:20.895687 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889389 4952 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 23:06:20.895687 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889392 4952 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 23:06:20.895687 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889396 4952 flags.go:64] FLAG: --enable-server="true" Oct 30 23:06:20.895687 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889399 4952 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 23:06:20.895687 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889404 4952 flags.go:64] FLAG: --event-burst="100" Oct 30 23:06:20.895687 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889407 4952 flags.go:64] FLAG: --event-qps="50" Oct 30 23:06:20.895687 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889410 4952 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 23:06:20.895687 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889413 4952 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 23:06:20.895687 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889416 4952 flags.go:64] FLAG: --eviction-hard="" Oct 30 23:06:20.896285 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889450 4952 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 23:06:20.896285 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889454 4952 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 23:06:20.896285 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889457 4952 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 23:06:20.896285 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889461 4952 flags.go:64] FLAG: --eviction-soft="" Oct 30 23:06:20.896285 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889464 4952 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 23:06:20.896285 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889467 4952 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 23:06:20.896285 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889471 4952 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 23:06:20.896285 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889474 4952 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 23:06:20.896285 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889477 4952 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 23:06:20.896285 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889480 4952 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 23:06:20.896285 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889483 4952 flags.go:64] FLAG: --feature-gates="" Oct 30 23:06:20.896285 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889487 4952 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 23:06:20.896285 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889490 4952 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 23:06:20.896285 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889494 4952 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 23:06:20.896285 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889497 4952 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 23:06:20.896285 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889500 4952 flags.go:64] FLAG: --healthz-port="10248" Oct 30 23:06:20.896285 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889504 4952 flags.go:64] FLAG: --help="false" Oct 30 23:06:20.896285 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889507 4952 flags.go:64] FLAG: --hostname-override="ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:06:20.896285 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889510 4952 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 23:06:20.896285 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889513 4952 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 23:06:20.896285 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889516 4952 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 23:06:20.896285 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889519 4952 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 23:06:20.896285 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889522 4952 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 23:06:20.896834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889525 4952 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 23:06:20.896834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889528 4952 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 23:06:20.896834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889533 4952 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 23:06:20.896834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889536 4952 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 23:06:20.896834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889539 4952 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 23:06:20.896834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889543 4952 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 23:06:20.896834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889545 4952 flags.go:64] FLAG: --kube-reserved="" Oct 30 23:06:20.896834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889548 4952 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 23:06:20.896834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889551 4952 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 23:06:20.896834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889554 4952 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 23:06:20.896834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889557 4952 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 23:06:20.896834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889560 4952 flags.go:64] FLAG: --lock-file="" Oct 30 23:06:20.896834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889563 4952 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 23:06:20.896834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889565 4952 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 23:06:20.896834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889568 4952 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 23:06:20.896834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889576 4952 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 23:06:20.896834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889579 4952 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 23:06:20.896834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889582 4952 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 23:06:20.896834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889585 4952 flags.go:64] FLAG: --logging-format="text" Oct 30 23:06:20.896834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889587 4952 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 23:06:20.896834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889591 4952 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 23:06:20.896834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889593 4952 flags.go:64] FLAG: --manifest-url="" Oct 30 23:06:20.896834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889596 4952 flags.go:64] FLAG: --manifest-url-header="" Oct 30 23:06:20.896834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889601 4952 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 23:06:20.896834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889604 4952 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 23:06:20.897528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889608 4952 flags.go:64] FLAG: --max-pods="110" Oct 30 23:06:20.897528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889611 4952 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 23:06:20.897528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889614 4952 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 23:06:20.897528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889617 4952 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 23:06:20.897528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889620 4952 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 23:06:20.897528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889623 4952 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 23:06:20.897528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889626 4952 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 23:06:20.897528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889629 4952 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 23:06:20.897528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889637 4952 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 23:06:20.897528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889641 4952 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 23:06:20.897528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889644 4952 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 23:06:20.897528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889647 4952 flags.go:64] FLAG: --pod-cidr="" Oct 30 23:06:20.897528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889650 4952 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 23:06:20.897528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889652 4952 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 23:06:20.897528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889655 4952 flags.go:64] FLAG: --pods-per-core="0" Oct 30 23:06:20.897528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889658 4952 flags.go:64] FLAG: --port="10250" Oct 30 23:06:20.897528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889661 4952 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 23:06:20.897528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889664 4952 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0ef590e33c2ab0754" Oct 30 23:06:20.897528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889668 4952 flags.go:64] FLAG: --qos-reserved="" Oct 30 23:06:20.897528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889671 4952 flags.go:64] FLAG: --read-only-port="10255" Oct 30 23:06:20.897528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889674 4952 flags.go:64] FLAG: --register-node="true" Oct 30 23:06:20.897528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889676 4952 flags.go:64] FLAG: --register-with-taints="" Oct 30 23:06:20.897528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889681 4952 flags.go:64] FLAG: --registry-burst="10" Oct 30 23:06:20.897528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889684 4952 flags.go:64] FLAG: --registry-qps="5" Oct 30 23:06:20.897528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889687 4952 flags.go:64] FLAG: --reserved-cpus="" Oct 30 23:06:20.897528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889690 4952 flags.go:64] FLAG: --reserved-memory="" Oct 30 23:06:20.898223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889694 4952 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 23:06:20.898223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889697 4952 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 23:06:20.898223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889700 4952 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 23:06:20.898223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889703 4952 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 23:06:20.898223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889706 4952 flags.go:64] FLAG: --runonce="false" Oct 30 23:06:20.898223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889709 4952 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 23:06:20.898223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889712 4952 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 23:06:20.898223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889715 4952 flags.go:64] FLAG: --seccomp-default="false" Oct 30 23:06:20.898223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889718 4952 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 23:06:20.898223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889721 4952 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 23:06:20.898223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889724 4952 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 23:06:20.898223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889727 4952 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 23:06:20.898223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889730 4952 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 23:06:20.898223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889733 4952 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 23:06:20.898223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889736 4952 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 23:06:20.898223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889740 4952 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 23:06:20.898223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889743 4952 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 23:06:20.898223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889746 4952 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 23:06:20.898223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889749 4952 flags.go:64] FLAG: --system-cgroups="" Oct 30 23:06:20.898223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889752 4952 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 23:06:20.898223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889757 4952 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 23:06:20.898223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889760 4952 flags.go:64] FLAG: --tls-cert-file="" Oct 30 23:06:20.898223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889763 4952 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 23:06:20.898223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889766 4952 flags.go:64] FLAG: --tls-min-version="" Oct 30 23:06:20.898223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889769 4952 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 23:06:20.898817 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889772 4952 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 23:06:20.898817 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889774 4952 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 23:06:20.898817 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889777 4952 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 23:06:20.898817 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889781 4952 flags.go:64] FLAG: --v="2" Oct 30 23:06:20.898817 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889786 4952 flags.go:64] FLAG: --version="false" Oct 30 23:06:20.898817 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889790 4952 flags.go:64] FLAG: --vmodule="" Oct 30 23:06:20.898817 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889793 4952 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 23:06:20.898817 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.889797 4952 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 23:06:20.898817 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889963 4952 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:06:20.898817 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889969 4952 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:06:20.898817 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889972 4952 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:06:20.898817 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889975 4952 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:06:20.898817 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889977 4952 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:06:20.898817 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889980 4952 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:06:20.898817 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889983 4952 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:06:20.898817 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889985 4952 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:06:20.898817 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889988 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:06:20.898817 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889991 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:06:20.898817 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889994 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:06:20.898817 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.889998 4952 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:06:20.898817 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890002 4952 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:06:20.899611 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890005 4952 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:06:20.899611 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890008 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:06:20.899611 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890011 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:06:20.899611 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890014 4952 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:06:20.899611 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890017 4952 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:06:20.899611 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890019 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:06:20.899611 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890022 4952 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:06:20.899611 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890024 4952 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:06:20.899611 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890027 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:06:20.899611 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890029 4952 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:06:20.899611 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890045 4952 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:06:20.899611 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890048 4952 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:06:20.899611 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890050 4952 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:06:20.899611 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890053 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:06:20.899611 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890057 4952 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:06:20.899611 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890060 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:06:20.899611 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890062 4952 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:06:20.899611 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890065 4952 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:06:20.899611 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890067 4952 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:06:20.899611 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890070 4952 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:06:20.900232 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890073 4952 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:06:20.900232 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890076 4952 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:06:20.900232 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890078 4952 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:06:20.900232 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890081 4952 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:06:20.900232 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890085 4952 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:06:20.900232 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890088 4952 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:06:20.900232 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890091 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:06:20.900232 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890094 4952 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:06:20.900232 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890099 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:06:20.900232 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890102 4952 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:06:20.900232 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890104 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:06:20.900232 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890107 4952 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:06:20.900232 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890110 4952 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:06:20.900232 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890112 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:06:20.900232 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890115 4952 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:06:20.900232 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890118 4952 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:06:20.900232 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890120 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:06:20.900232 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890123 4952 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:06:20.900232 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890125 4952 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:06:20.901004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890128 4952 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:06:20.901004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890131 4952 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:06:20.901004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890133 4952 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:06:20.901004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890136 4952 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:06:20.901004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890138 4952 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:06:20.901004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890141 4952 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:06:20.901004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890143 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:06:20.901004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890147 4952 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:06:20.901004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890150 4952 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:06:20.901004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890152 4952 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:06:20.901004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890155 4952 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:06:20.901004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890157 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:06:20.901004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890160 4952 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:06:20.901004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890162 4952 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:06:20.901004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890165 4952 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:06:20.901004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890167 4952 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:06:20.901004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890170 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:06:20.901004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890173 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:06:20.901004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890175 4952 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:06:20.901004 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890178 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:06:20.901683 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890180 4952 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:06:20.901683 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890183 4952 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:06:20.901683 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890192 4952 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:06:20.901683 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890195 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:06:20.901683 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890198 4952 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:06:20.901683 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890200 4952 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:06:20.901683 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890203 4952 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:06:20.901683 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890205 4952 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:06:20.901683 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890208 4952 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:06:20.901683 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890210 4952 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:06:20.901683 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890213 4952 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:06:20.901683 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890216 4952 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:06:20.901683 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890218 4952 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:06:20.901683 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890221 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:06:20.901683 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890223 4952 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:06:20.901683 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890226 4952 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:06:20.901683 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890228 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:06:20.901683 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890231 4952 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:06:20.901683 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890233 4952 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:06:20.901683 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890237 4952 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:06:20.902260 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890240 4952 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:06:20.902260 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890242 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:06:20.902260 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890245 4952 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:06:20.902260 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890248 4952 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:06:20.902260 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890250 4952 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:06:20.902260 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890253 4952 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:06:20.902260 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890255 4952 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:06:20.902260 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890258 4952 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:06:20.902260 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890261 4952 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:06:20.902260 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890263 4952 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:06:20.902260 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890266 4952 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:06:20.902260 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890268 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:06:20.902260 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890271 4952 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:06:20.902260 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890273 4952 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:06:20.902260 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.890300 4952 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:06:20.902260 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890348 4952 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:06:20.902675 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890353 4952 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:06:20.902675 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890356 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:06:20.902675 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890359 4952 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:06:20.902675 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890362 4952 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:06:20.902675 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890364 4952 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:06:20.902675 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890367 4952 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:06:20.902675 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890369 4952 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:06:20.902675 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890372 4952 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:06:20.902675 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890375 4952 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:06:20.902675 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890377 4952 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:06:20.902675 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890380 4952 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:06:20.902675 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890383 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:06:20.902675 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890385 4952 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:06:20.902675 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890388 4952 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:06:20.902675 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890390 4952 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:06:20.902675 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890394 4952 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:06:20.902675 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890397 4952 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:06:20.902675 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890399 4952 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:06:20.902675 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890402 4952 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:06:20.902675 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890406 4952 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:06:20.903339 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890408 4952 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:06:20.903339 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890411 4952 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:06:20.903339 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890413 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:06:20.903339 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890416 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:06:20.903339 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890419 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:06:20.903339 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890422 4952 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:06:20.903339 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890426 4952 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:06:20.903339 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890429 4952 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:06:20.903339 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890433 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:06:20.903339 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890435 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:06:20.903339 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890438 4952 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:06:20.903339 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890440 4952 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:06:20.903339 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890449 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:06:20.903339 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890451 4952 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:06:20.903339 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890454 4952 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:06:20.903339 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890457 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:06:20.903339 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890459 4952 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:06:20.903339 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890462 4952 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:06:20.903339 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890464 4952 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:06:20.903926 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890467 4952 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:06:20.903926 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890469 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:06:20.903926 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890472 4952 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:06:20.903926 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890474 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:06:20.903926 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890477 4952 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:06:20.903926 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890480 4952 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:06:20.903926 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890482 4952 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:06:20.903926 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890485 4952 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:06:20.903926 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890489 4952 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:06:20.903926 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890492 4952 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:06:20.903926 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890495 4952 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:06:20.903926 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890498 4952 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:06:20.903926 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890503 4952 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:06:20.903926 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890506 4952 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:06:20.903926 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890509 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:06:20.903926 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890511 4952 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:06:20.903926 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890514 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:06:20.903926 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890517 4952 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:06:20.903926 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890520 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:06:20.904477 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890522 4952 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:06:20.904477 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890525 4952 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:06:20.904477 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890528 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:06:20.904477 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890530 4952 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:06:20.904477 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890533 4952 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:06:20.904477 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890535 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:06:20.904477 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890538 4952 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:06:20.904477 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890546 4952 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:06:20.904477 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890548 4952 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:06:20.904477 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890552 4952 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:06:20.904477 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890554 4952 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:06:20.904477 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890557 4952 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:06:20.904477 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890559 4952 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:06:20.904477 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890562 4952 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:06:20.904477 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890565 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:06:20.904477 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890567 4952 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:06:20.904477 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890570 4952 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:06:20.904477 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890572 4952 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:06:20.904477 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890574 4952 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:06:20.904477 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890577 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:06:20.905095 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890579 4952 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:06:20.905095 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890584 4952 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:06:20.905095 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890586 4952 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:06:20.905095 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890589 4952 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:06:20.905095 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890592 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:06:20.905095 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890596 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:06:20.905095 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890599 4952 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:06:20.905095 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890601 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:06:20.905095 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890604 4952 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:06:20.905095 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890606 4952 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:06:20.905095 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890609 4952 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:06:20.905095 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890612 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:06:20.905095 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890615 4952 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:06:20.905095 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890617 4952 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:06:20.905095 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890620 4952 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:06:20.905095 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890622 4952 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:06:20.905095 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890625 4952 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:06:20.905095 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890628 4952 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:06:20.905095 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890630 4952 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:06:20.905095 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890633 4952 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:06:20.905585 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890636 4952 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:06:20.905585 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890640 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:06:20.905585 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890643 4952 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:06:20.905585 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890646 4952 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:06:20.905585 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890648 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:06:20.905585 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890651 4952 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:06:20.905585 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.890653 4952 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:06:20.905585 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.894517 4952 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 23:06:20.905585 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.894530 4952 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 23:06:20.905585 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.894548 4952 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 23:06:20.905585 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.894553 4952 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 23:06:20.905585 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894635 4952 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:06:20.905585 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894641 4952 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:06:20.905585 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894644 4952 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:06:20.905585 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894647 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:06:20.905585 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894649 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:06:20.905585 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894652 4952 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:06:20.905585 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894655 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:06:20.905585 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894658 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:06:20.906082 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894661 4952 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:06:20.906082 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894663 4952 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:06:20.906082 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894666 4952 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:06:20.906082 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894669 4952 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:06:20.906082 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894672 4952 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:06:20.906082 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894675 4952 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:06:20.906082 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894678 4952 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:06:20.906082 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894680 4952 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:06:20.906082 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894684 4952 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:06:20.906082 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894688 4952 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:06:20.906082 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894691 4952 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:06:20.906082 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894694 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:06:20.906082 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894696 4952 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:06:20.906082 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894700 4952 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:06:20.906082 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894703 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:06:20.906082 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894705 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:06:20.906082 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894708 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:06:20.906082 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894711 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:06:20.906082 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894713 4952 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:06:20.906082 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894716 4952 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:06:20.906572 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894719 4952 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:06:20.906572 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894721 4952 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:06:20.906572 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894724 4952 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:06:20.906572 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894727 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:06:20.906572 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894729 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:06:20.906572 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894732 4952 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:06:20.906572 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894734 4952 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:06:20.906572 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894737 4952 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:06:20.906572 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894740 4952 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:06:20.906572 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894742 4952 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:06:20.906572 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894745 4952 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:06:20.906572 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894748 4952 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:06:20.906572 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894750 4952 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:06:20.906572 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894753 4952 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:06:20.906572 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894755 4952 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:06:20.906572 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894758 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:06:20.906572 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894761 4952 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:06:20.906572 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894763 4952 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:06:20.906572 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894766 4952 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:06:20.907079 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894769 4952 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:06:20.907079 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894771 4952 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:06:20.907079 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894774 4952 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:06:20.907079 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894776 4952 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:06:20.907079 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894779 4952 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:06:20.907079 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894781 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:06:20.907079 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894784 4952 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:06:20.907079 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894793 4952 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:06:20.907079 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894796 4952 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:06:20.907079 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894799 4952 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:06:20.907079 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894801 4952 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:06:20.907079 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894804 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:06:20.907079 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894806 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:06:20.907079 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894808 4952 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:06:20.907079 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894811 4952 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:06:20.907079 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894814 4952 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:06:20.907079 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894816 4952 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:06:20.907079 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894819 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:06:20.907079 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894821 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:06:20.907540 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894824 4952 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:06:20.907540 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894826 4952 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:06:20.907540 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894829 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:06:20.907540 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894831 4952 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:06:20.907540 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894834 4952 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:06:20.907540 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894837 4952 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:06:20.907540 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894839 4952 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:06:20.907540 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894842 4952 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:06:20.907540 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894844 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:06:20.907540 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894847 4952 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:06:20.907540 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894849 4952 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:06:20.907540 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894852 4952 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:06:20.907540 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894854 4952 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:06:20.907540 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894857 4952 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:06:20.907540 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894859 4952 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:06:20.907540 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894862 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:06:20.907540 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894864 4952 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:06:20.907540 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894867 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:06:20.907540 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894869 4952 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:06:20.907540 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894872 4952 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:06:20.908050 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894874 4952 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:06:20.908050 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894883 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:06:20.908050 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894886 4952 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:06:20.908050 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894889 4952 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:06:20.908050 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894891 4952 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:06:20.908050 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894894 4952 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:06:20.908050 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894896 4952 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:06:20.908050 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894900 4952 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:06:20.908050 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894905 4952 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:06:20.908050 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894908 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:06:20.908050 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894911 4952 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:06:20.908050 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894914 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:06:20.908050 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894917 4952 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:06:20.908050 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894920 4952 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:06:20.908050 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894922 4952 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:06:20.908050 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894925 4952 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:06:20.908050 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894927 4952 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:06:20.908050 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894929 4952 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:06:20.908050 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894932 4952 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:06:20.908050 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.894935 4952 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:06:20.908530 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.894961 4952 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:06:20.908530 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895159 4952 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:06:20.908530 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895164 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:06:20.908530 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895167 4952 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:06:20.908530 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895170 4952 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:06:20.908530 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895173 4952 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:06:20.908530 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895175 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:06:20.908530 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895178 4952 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:06:20.908530 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895180 4952 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:06:20.908530 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895184 4952 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:06:20.908530 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895189 4952 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:06:20.908530 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895191 4952 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:06:20.908530 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895195 4952 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:06:20.908530 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895197 4952 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:06:20.908530 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895205 4952 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:06:20.908530 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895208 4952 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:06:20.908930 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895211 4952 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:06:20.908930 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895213 4952 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:06:20.908930 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895216 4952 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:06:20.908930 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895218 4952 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:06:20.908930 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895221 4952 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:06:20.908930 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895223 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:06:20.908930 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895226 4952 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:06:20.908930 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895228 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:06:20.908930 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895231 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:06:20.908930 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895233 4952 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:06:20.908930 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895236 4952 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:06:20.908930 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895238 4952 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:06:20.908930 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895241 4952 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:06:20.908930 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895243 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:06:20.908930 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895246 4952 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:06:20.908930 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895248 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:06:20.908930 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895251 4952 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:06:20.908930 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895253 4952 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:06:20.908930 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895256 4952 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:06:20.908930 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895258 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:06:20.909552 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895261 4952 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:06:20.909552 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895263 4952 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:06:20.909552 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895266 4952 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:06:20.909552 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895268 4952 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:06:20.909552 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895271 4952 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:06:20.909552 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895273 4952 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:06:20.909552 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895276 4952 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:06:20.909552 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895278 4952 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:06:20.909552 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895281 4952 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:06:20.909552 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895284 4952 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:06:20.909552 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895286 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:06:20.909552 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895294 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:06:20.909552 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895297 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:06:20.909552 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895300 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:06:20.909552 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895302 4952 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:06:20.909552 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895305 4952 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:06:20.909552 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895308 4952 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:06:20.909552 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895310 4952 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:06:20.909552 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895313 4952 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:06:20.909552 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895315 4952 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:06:20.910062 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895318 4952 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:06:20.910062 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895321 4952 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:06:20.910062 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895323 4952 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:06:20.910062 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895326 4952 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:06:20.910062 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895328 4952 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:06:20.910062 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895331 4952 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:06:20.910062 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895333 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:06:20.910062 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895336 4952 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:06:20.910062 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895338 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:06:20.910062 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895341 4952 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:06:20.910062 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895343 4952 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:06:20.910062 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895345 4952 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:06:20.910062 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895348 4952 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:06:20.910062 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895351 4952 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:06:20.910062 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895353 4952 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:06:20.910062 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895356 4952 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:06:20.910062 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895358 4952 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:06:20.910062 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895361 4952 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:06:20.910062 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895363 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:06:20.910575 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895366 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:06:20.910575 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895368 4952 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:06:20.910575 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895371 4952 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:06:20.910575 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895373 4952 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:06:20.910575 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895376 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:06:20.910575 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895384 4952 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:06:20.910575 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895386 4952 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:06:20.910575 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895389 4952 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:06:20.910575 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895392 4952 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:06:20.910575 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895394 4952 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:06:20.910575 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895397 4952 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:06:20.910575 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895401 4952 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:06:20.910575 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895404 4952 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:06:20.910575 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895407 4952 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:06:20.910575 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895410 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:06:20.910575 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895413 4952 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:06:20.910575 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895415 4952 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:06:20.910575 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895418 4952 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:06:20.910575 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895420 4952 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:06:20.910575 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895423 4952 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:06:20.911107 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895425 4952 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:06:20.911107 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895427 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:06:20.911107 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895430 4952 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:06:20.911107 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895432 4952 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:06:20.911107 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895435 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:06:20.911107 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895437 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:06:20.911107 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895440 4952 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:06:20.911107 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895443 4952 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:06:20.911107 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895445 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:06:20.911107 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895448 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:06:20.911107 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895453 4952 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:06:20.911107 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895455 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:06:20.911107 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.895479 4952 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:06:20.911107 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895525 4952 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:06:20.911107 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895529 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:06:20.911476 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895533 4952 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:06:20.911476 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895535 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:06:20.911476 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895538 4952 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:06:20.911476 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895546 4952 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:06:20.911476 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895549 4952 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:06:20.911476 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895552 4952 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:06:20.911476 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895554 4952 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:06:20.911476 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895557 4952 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:06:20.911476 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895559 4952 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:06:20.911476 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895562 4952 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:06:20.911476 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895565 4952 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:06:20.911476 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895567 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:06:20.911476 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895570 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:06:20.911476 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895572 4952 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:06:20.911476 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895575 4952 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:06:20.911476 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895577 4952 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:06:20.911476 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895580 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:06:20.911476 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895582 4952 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:06:20.911476 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895587 4952 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:06:20.911476 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895590 4952 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:06:20.911946 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895592 4952 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:06:20.911946 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895595 4952 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:06:20.911946 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895597 4952 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:06:20.911946 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895601 4952 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:06:20.911946 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895605 4952 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:06:20.911946 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895608 4952 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:06:20.911946 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895610 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:06:20.911946 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895613 4952 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:06:20.911946 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895617 4952 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:06:20.911946 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895620 4952 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:06:20.911946 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895622 4952 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:06:20.911946 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895625 4952 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:06:20.911946 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895627 4952 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:06:20.911946 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895630 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:06:20.911946 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895632 4952 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:06:20.911946 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895635 4952 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:06:20.911946 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895645 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:06:20.911946 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895648 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:06:20.911946 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895651 4952 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:06:20.912486 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895654 4952 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:06:20.912486 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895656 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:06:20.912486 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895659 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:06:20.912486 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895661 4952 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:06:20.912486 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895664 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:06:20.912486 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895666 4952 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:06:20.912486 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895669 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:06:20.912486 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895671 4952 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:06:20.912486 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895674 4952 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:06:20.912486 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895677 4952 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:06:20.912486 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895679 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:06:20.912486 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895683 4952 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:06:20.912486 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895686 4952 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:06:20.912486 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895690 4952 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:06:20.912486 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895694 4952 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:06:20.912486 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895697 4952 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:06:20.912486 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895700 4952 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:06:20.912486 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895703 4952 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:06:20.912486 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895705 4952 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:06:20.912951 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895708 4952 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:06:20.912951 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895710 4952 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:06:20.912951 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895714 4952 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:06:20.912951 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895717 4952 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:06:20.912951 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895720 4952 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:06:20.912951 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895722 4952 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:06:20.912951 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895725 4952 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:06:20.912951 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895727 4952 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:06:20.912951 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895730 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:06:20.912951 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895732 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:06:20.912951 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895735 4952 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:06:20.912951 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895743 4952 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:06:20.912951 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895746 4952 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:06:20.912951 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895748 4952 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:06:20.912951 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895751 4952 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:06:20.912951 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895754 4952 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:06:20.912951 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895756 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:06:20.912951 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895759 4952 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:06:20.912951 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895761 4952 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:06:20.912951 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895764 4952 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:06:20.913547 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895766 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:06:20.913547 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895769 4952 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:06:20.913547 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895771 4952 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:06:20.913547 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895774 4952 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:06:20.913547 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895777 4952 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:06:20.913547 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895780 4952 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:06:20.913547 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895783 4952 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:06:20.913547 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895785 4952 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:06:20.913547 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895788 4952 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:06:20.913547 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895790 4952 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:06:20.913547 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895793 4952 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:06:20.913547 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895796 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:06:20.913547 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895798 4952 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:06:20.913547 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895800 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:06:20.913547 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895804 4952 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:06:20.913547 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895807 4952 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:06:20.913547 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895810 4952 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:06:20.913547 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895812 4952 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:06:20.913547 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895815 4952 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:06:20.913547 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895818 4952 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:06:20.914102 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895820 4952 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:06:20.914102 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895823 4952 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:06:20.914102 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895830 4952 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:06:20.914102 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895832 4952 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:06:20.914102 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895840 4952 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:06:20.914102 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:20.895843 4952 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:06:20.914102 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.895891 4952 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 23:06:20.914102 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.896854 4952 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 23:06:20.914102 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.896914 4952 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 23:06:20.914102 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.897121 4952 server.go:1014] "Starting client certificate rotation" Oct 30 23:06:20.914102 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.897313 4952 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 23:06:20.914102 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.897418 4952 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 23:00:38 +0000 UTC" deadline="2028-06-22 12:37:09.250258201 +0000 UTC" Oct 30 23:06:20.914102 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.897482 4952 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14413h30m48.352778041s" Oct 30 23:06:20.914102 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.897695 4952 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 23:06:20.914102 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.898942 4952 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 23:06:20.914102 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.901253 4952 log.go:25] "Validated CRI v1 runtime API" Oct 30 23:06:20.914496 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.903364 4952 log.go:25] "Validated CRI v1 image API" Oct 30 23:06:20.914496 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.904114 4952 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 23:06:20.914496 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.905931 4952 fs.go:135] Filesystem UUIDs: map[0e92ea7b-a59c-49db-9b10-7675b5623f00:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c20af26b-c93f-486e-bf9e-32c773c4c313:/dev/nvme0n1p3] Oct 30 23:06:20.914592 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.905948 4952 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0e986cf43acdc91e04b55a1643afd28a907823e006cc7c87146f051e1aa17524/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0e986cf43acdc91e04b55a1643afd28a907823e006cc7c87146f051e1aa17524/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4374a26c15def0481cf21509e6531d6926bb66c03e0edd6e8e5b64fd66fc22e9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4374a26c15def0481cf21509e6531d6926bb66c03e0edd6e8e5b64fd66fc22e9/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/487de86185e608fc2799e6118eb17391576d26d64e37d4d9ef3ae300dbe115d0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/487de86185e608fc2799e6118eb17391576d26d64e37d4d9ef3ae300dbe115d0/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4dd83f85253bebe12c121175d879cd3fd7e9e87a39ffe8f058ac37eee18f8c9c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4dd83f85253bebe12c121175d879cd3fd7e9e87a39ffe8f058ac37eee18f8c9c/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/622a3003f71c337888542d7ca03cfeea3d5ba41630ac893ca1b000a06ddd87b9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/622a3003f71c337888542d7ca03cfeea3d5ba41630ac893ca1b000a06ddd87b9/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/724f369db6e1b26e89582a68d830fd9f8b170a494d63ea2253ef416049518117/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/724f369db6e1b26e89582a68d830fd9f8b170a494d63ea2253ef416049518117/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8cc1b5daf74f5223ffb2285e88662648c93d1d807a8cdc3775c9a4b57ec74333/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8cc1b5daf74f5223ffb2285e88662648c93d1d807a8cdc3775c9a4b57ec74333/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8fe7a5b2e0ae165f7854fb568f1b3b75bc57b21564b5d529b3a2076811511adc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8fe7a5b2e0ae165f7854fb568f1b3b75bc57b21564b5d529b3a2076811511adc/userdata/shm major:0 minor:252 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/93ca6840cfd406c206a6093b26ff3ae4d0b6daef2ba2a11b345def032d54cd6f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/93ca6840cfd406c206a6093b26ff3ae4d0b6daef2ba2a11b345def032d54cd6f/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e0f56088f54a7f4479099ef4bd5f80a2b25ff5657f33c8920d01d9857369059/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e0f56088f54a7f4479099ef4bd5f80a2b25ff5657f33c8920d01d9857369059/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e5aeafe6cf21dbb40184cceedf15ae29eb2517a993918361607d6e3f5dde5fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e5aeafe6cf21dbb40184cceedf15ae29eb2517a993918361607d6e3f5dde5fb/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a2e52979c7c424c66dfef4b2a3bb8bd8cb031109f65b823fb2725c331b66d839/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a2e52979c7c424c66dfef4b2a3bb8bd8cb031109f65b823fb2725c331b66d839/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a959ec9ff3ac805313fedb7b37a657941adadffaec8ffc81bace9e5ebc780d02/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a959ec9ff3ac805313fedb7b37a657941adadffaec8ffc81bace9e5ebc780d02/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b71dbcb555e71fae6aaffc5d8c19a7adc8dead3a2e5a786976813e1af918144a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b71dbcb555e71fae6aaffc5d8c19a7adc8dead3a2e5a786976813e1af918144a/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c249bb339a7fff2e18422f1f388959232c4d30a100cdd65db5dfd96c0da2140b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c249bb339a7fff2e18422f1f388959232c4d30a100cdd65db5dfd96c0da2140b/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c835c85a86c44cbbcd65500910f97f39ea2a458b5919bacf36b6eb4485c9ef4b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c835c85a86c44cbbcd65500910f97f39ea2a458b5919bacf36b6eb4485c9ef4b/userdata/shm major:0 minor:248 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c9afe92d8ad57aa5fdeba3b2c942001aeb12544434f72c6727b33f2f0802d26d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c9afe92d8ad57aa5fdeba3b2c942001aeb12544434f72c6727b33f2f0802d26d/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ce66ace81ace169fbc0c516afacbdf9d1e4f43fb8bff281b5ae5dcfacf220b1d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ce66ace81ace169fbc0c516afacbdf9d1e4f43fb8bff281b5ae5dcfacf220b1d/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d57a493fc14ecfecab1a187940f32a6090599edd683d5d078a4f75bb8e8846ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d57a493fc14ecfecab1a187940f32a6090599edd683d5d078a4f75bb8e8846ad/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dd4fa69b3d38d1465c0b30f5f395947b0ff1d833244a71f20b8924d08ad75ed9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dd4fa69b3d38d1465c0b30f5f395947b0ff1d833244a71f20b8924d08ad75ed9/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/df5f40c01fa381a557e544f6888feca5bb2fe273560e70b3fa5669ae9a73ab02/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/df5f40c01fa381a557e544f6888feca5bb2fe273560e70b3fa5669ae9a73ab02/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e50f5624186bdf7c4bb7e28866f82676e6888120749c7b1056e4e72297fb3b6a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e50f5624186bdf7c4bb7e28866f82676e6888120749c7b1056e4e72297fb3b6a/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e5deb16a572b4a313257ac5667bbc9a6f23ad2e38dfc6313dbf21bf25c868847/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e5deb16a572b4a313257ac5667bbc9a6f23ad2e38dfc6313dbf21bf25c868847/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1cc88905-e18f-4a2d-a503-4b4950cd27d3/volumes/kubernetes.io~projected/kube-api-access-q42l2:{mountpoint:/var/lib/kubelet/pods/1cc88905-e18f-4a2d-a503-4b4950cd27d3/volumes/kubernetes.io~projected/kube-api-access-q42l2 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/volumes/kubernetes.io~projected/kube-api-access-b4cs6:{mountpoint:/var/lib/kubelet/pods/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/volumes/kubernetes.io~projected/kube-api-access-b4cs6 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2861d402-9b72-4907-b8c1-7199ee3a8e92/volumes/kubernetes.io~projected/kube-api-access-pc5qz:{mountpoint:/var/lib/kubelet/pods/2861d402-9b72-4907-b8c1-7199ee3a8e92/volumes/kubernetes.io~projected/kube-api-access-pc5qz major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/322e3a9b-911f-4cd8-8f43-71d56152e917/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/322e3a9b-911f-4cd8-8f43-71d56152e917/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3838326a-857d-4511-8866-813015d85554/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/3838326a-857d-4511-8866-813015d85554/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3838326a-857d-4511-8866-813015d85554/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/3838326a-857d-4511-8866-813015d85554/volumes/kubernetes.io~empty-dir/tmp major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3838326a-857d-4511-8866-813015d85554/volumes/kubernetes.io~projected/kube-api-access-rjr2r:{mountpoint:/var/lib/kubelet/pods/3838326a-857d-4511-8866-813015d85554/volumes/kubernetes.io~projected/kube-api-access-rjr2r major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ec44853-19be-4bbb-9840-bb4c96fd2a29/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/3ec44853-19be-4bbb-9840-bb4c96fd2a29/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ec44853-19be-4bbb-9840-bb4c96fd2a29/volumes/kubernetes.io~projected/kube-api-access-pvnfz:{mountpoint:/var/lib/kubelet/pods/3ec44853-19be-4bbb-9840-bb4c96fd2a29/volumes/kubernetes.io~projected/kube-api-access-pvnfz major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ec44853-19be-4bbb-9840-bb4c96fd2a29/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/3ec44853-19be-4bbb-9840-bb4c96fd2a29/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ec44853-19be-4bbb-9840-bb4c96fd2a29/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/3ec44853-19be-4bbb-9840-bb4c96fd2a29/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f7d22b5-256e-47d8-947f-b89b47f14d08/volumes/kubernetes.io~projected/kube-api-access-l9nfl:{mountpoint:/var/lib/kubelet/pods/4f7d22b5-256e-47d8-947f-b89b47f14d08/volumes/kubernetes.io~projected/kube-api-access-l9nfl major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/582d0663-6eba-4e69-81b0-fb25380f4c47/volumes/kubernetes.io~projected/kube-api-access-vg5n4:{mountpoint:/var/lib/kubelet/pods/582d0663-6eba-4e69-81b0-fb25380f4c47/volumes/kubernetes.io~projected/kube-api-access-vg5n4 major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/582d0663-6eba-4e69-81b0-fb25380f4c47/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/582d0663-6eba-4e69-81b0-fb25380f4c47/volumes/kubernetes.io~secret/cert major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58d18552-3df9-47f4-9291-b0e82cd954d2/volumes/kubernetes.io~projected/kube-api-access-zvlmc:{mountpoint:/var/lib/kubelet/pods/58d18552-3df9-47f4-9291-b0e82cd954d2/volumes/kubernetes.io~projected/kube-api-access-zvlmc major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5bfb1046-f8b5-4035-bb35-173a1316ea0f/volumes/kubernetes.io~projected/kube-api-access-lftl4:{mountpoint:/var/lib/kubelet/pods/5bfb1046-f8b5-4035-bb35-173a1316ea0f/volumes/kubernetes.io~projected/kube-api-access-lftl4 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5cd7decf-5213-4360-a405-76d07c6ae3df/volumes/kubernetes.io~projected/kube-api-access-9tm5m:{mountpoint:/var/lib/kubelet/pods/5cd7decf-5213-4360-a405-76d07c6ae3df/volumes/kubernetes.io~projected/kube-api-access-9tm5m major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f6c1ad7-10c4-4995-ae18-315d38184360/volumes/kubernetes.io~projected/kube-api-access-8czng:{mountpoint:/var/lib/kubelet/pods/5f6c1ad7-10c4-4995-ae18-315d38184360/volumes/kubernetes.io~projected/kube-api-access-8czng major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f6c1ad7-10c4-4995-ae18-315d38184360/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/5f6c1ad7-10c4-4995-ae18-315d38184360/volumes/kubernetes.io~secret/default-certificate major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f6c1ad7-10c4-4995-ae18-315d38184360/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/5f6c1ad7-10c4-4995-ae18-315d38184360/volumes/kubernetes.io~secret/stats-auth major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/601f3571-aa13-43c6-8b5a-393029720fc3/volumes/kubernetes.io~projected/kube-api-access-rdsrp:{mountpoint:/var/lib/kubelet/pods/601f3571-aa13-43c6-8b5a-393029720fc3/volumes/kubernetes.io~projected/kube-api-access-rdsrp major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/624d7fe2-4960-48ba-b711-3d85e4bf0687/volumes/kubernetes.io~projected/kube-api-access-gcls9:{mountpoint:/var/lib/kubelet/pods/624d7fe2-4960-48ba-b711-3d85e4bf0687/volumes/kubernetes.io~projected/kube-api-access-gcls9 major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67e30e6a-a537-4e2c-a484-f15281828fec/volumes/kubernetes.io~projected/kube-api-access-b754k:{mountpoint:/var/lib/kubelet/pods/67e30e6a-a537-4e2c-a484-f15281828fec/volumes/kubernetes.io~projected/kube-api-access-b754k major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893/volumes/kubernetes.io~projected/kube-api-access-vvh4w:{mountpoint:/var/lib/kubelet/pods/6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893/volumes/kubernetes.io~projected/kube-api-access-vvh4w major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/706016da-3413-43e2-b06e-944572ca4106/volumes/kubernetes.io~projected/kube-api-access-7t9v8:{mountpoint:/var/lib/kubelet/pods/706016da-3413-43e2-b06e-944572ca4106/volumes/kubernetes.io~projected/kube-api-access-7t9v8 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5/volumes/kubernetes.io~projected/kube-api-access-g48ld:{mountpoint:/var/lib/kubelet/pods/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5/volumes/kubernetes.io~projected/kube-api-access-g48ld major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a7f82b8-115d-43ed-a7a2-cb869a7e48e0/volumes/kubernetes.io~projected/kube-api-access-rtk5p:{mountpoint:/var/lib/kubelet/pods/8a7f82b8-115d-43ed-a7a2-cb869a7e48e0/volumes/kubernetes.io~projected/kube-api-access-rtk5p major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cbb1813-35e1-49cb-8718-d0ba47d2350d/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/8cbb1813-35e1-49cb-8718-d0ba47d2350d/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cbb1813-35e1-49cb-8718-d0ba47d2350d/volumes/kubernetes.io~projected/kube-api-access-tpjxg:{mountpoint:/var/lib/kubelet/pods/8cbb1813-35e1-49cb-8718-d0ba47d2350d/volumes/kubernetes.io~projected/kube-api-access-tpjxg major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cbb1813-35e1-49cb-8718-d0ba47d2350d/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/8cbb1813-35e1-49cb-8718-d0ba47d2350d/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cbb1813-35e1-49cb-8718-d0ba47d2350d/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/8cbb1813-35e1-49cb-8718-d0ba47d2350d/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad111ef6-4d18-4438-a0cc-e31a46804710/volumes/kubernetes.io~projected/kube-api-access-ldxjv:{mountpoint:/var/lib/kubelet/pods/ad111ef6-4d18-4438-a0cc-e31a46804710/volumes/kubernetes.io~projected/kube-api-access-ldxjv major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcddee82-a2d2-471b-badb-0295456e6a90/volumes/kubernetes.io~projected/kube-api-access-4b6bz:{mountpoint:/var/lib/kubelet/pods/bcddee82-a2d2-471b-badb-0295456e6a90/volumes/kubernetes.io~projected/kube-api-access-4b6bz major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4368706-761a-4f6e-9dda-a8087a5f4fdc/volumes/kubernetes.io~projected/kube-api-access-qtz5v:{mountpoint:/var/lib/kubelet/pods/c4368706-761a-4f6e-9dda-a8087a5f4fdc/volumes/kubernetes.io~projected/kube-api-access-qtz5v major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dec2c5fa-4b9a-47ed-b005-04874761f419/volumes/kubernetes.io~projected/kube-api-access-r9g9l:{mountpoint:/var/lib/kubelet/pods/dec2c5fa-4b9a-47ed-b005-04874761f419/volumes/kubernetes.io~projected/kube-api-access-r9g9l major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e32c5dc1-43ac-4ca5-a391-947b68d10d51/volumes/kubernetes.io~projected/kube-api-access-xp7ls:{mountpoint:/var/lib/kubelet/pods/e32c5dc1-43ac-4ca5-a391-947b68d10d51/volumes/kubernetes.io~projected/kube-api-access-xp7ls major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e4d82728-b04b-4914-93d2-fce161acae36/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/e4d82728-b04b-4914-93d2-fce161acae36/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9a89b14-a8e8-45f2-84bd-dba21617499c/volumes/kubernetes.io~projected/kube-api-access-fc7nh:{mountpoint:/var/lib/kubelet/pods/e9a89b14-a8e8-45f2-84bd-dba21617499c/volumes/kubernetes.io~projected/kube-api-access-fc7nh major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9a89b14-a8e8-45f2-84bd-dba21617499c/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/e9a89b14-a8e8-45f2-84bd-dba21617499c/volumes/kubernetes.io~secret/serving-cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/edc2af65-1515-4742-89e1-b49814016b09/volumes/kubernetes.io~projected/kube-api-access-6bxrl:{mountpoint:/var/lib/kubelet/pods/edc2af65-1515-4742-89e1-b49814016b09/volumes/kubernetes.io~projected/kube-api-access-6bxrl major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/edc2af65-1515-4742-89e1-b49814016b09/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/edc2af65-1515-4742-89e1-b49814016b09/volumes/kubernetes.io~secret/serving-cert major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef598d5f-a066-4b2d-96bd-fbfc47741dbb/volumes/kubernetes.io~projected/kube-api-access-kl67b:{mountpoint:/var/lib/kubelet/pods/ef598d5f-a066-4b2d-96bd-fbfc47741dbb/volumes/kubernetes.io~projected/kube-api-access-kl67b major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef598d5f-a066-4b2d-96bd-fbfc47741dbb/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/ef598d5f-a066-4b2d-96bd-fbfc47741dbb/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f644db3c-e78a-42a8-9af3-c61194a21839/volumes/kubernetes.io~projected/kube-api-access-qbm49:{mountpoint:/var/lib/kubelet/pods/f644db3c-e78a-42a8-9af3-c61194a21839/volumes/kubernetes.io~projected/kube-api-access-qbm49 major:0 minor:218 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/fe3fdc85bc275857a8480471c044ace234c8dc5a9f17d17cb556e92a388cbb98/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/f5bd2808fd55d80e180f5b8d8f5689f38eef25c338062e41c9cb9556e4565423/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/e55d37f479e085a9cd24c56df6e08c89034e543f4f4b4f1007282763327e3e5e/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/f8827d0da2902e18a2820e4646a845db9c7bd4b40c9c5fb331566527e87fd660/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/02c0691a3438b8626f0ecec4b3237560b7daf6f57679ff950ce2712c6dc88929/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/4c4e0f0d797ba1b726c51e3b7c23cc30e8d2ec625867606157b2385fe7738c4d/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/36a32eeb1c867a2dd8dc176b29f519a4e7e2c06b1d3a32baf74d9e44c7620384/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/6419079b04bb17aa0d39822311f42b2508c26e31443e53629fd1d1e28d60c995/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/2a2868a00b9012215845b464a7ca4dbe45e2f24c0932d2bd31366ad1d6b775af/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/9aaea94bd0b8a951ede7aa47cde30cd3c5cc29ff394e788cbbe0997090d7ec6a/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/e472a99bad6d0efd6dcd2d8a16e1520b6ab8f3daf8d9c3d0ebb1b76eba60fb73/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/b23b4eecb3923c0a60f8ea4f7ae75afae275c2fa4a8eb1d48f8145eac99d3f4d/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/926bca4a61d069544e6092a6f9343e943c0978e7ae717cd79cab411f6fb01136/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/dbf3c377219bab939d7de47ced870f283b7a2fb42ac5d2eef2b03ecf7c058756/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/c19133ffcab12ba52e50b8177bcb9e12d967851f388132fdbfa4f7b9761aad88/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/8edf2f12ab190c93af5a259ffaea3d56f8a0032de2f169ebacefb7ca342ca44b/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/31a51432be22fe53041fcc593756ec1617d5de0a672c63e8b1caf98c08c2aaea/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/e91ffedf9ccd15aec07eba0b04746692859d74b7e18680aa2a25e983adefac3e/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/bb61b7be61f6a5d4ccce6ced4e7426f40b8f49d9aebffb6215535a2794445409/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-186:{mountpoint:/var/lib/containers/storage/overlay/3d5568fb21a5266477167fa6100aeb8308ca43beab2a205378abcdd8dbb028e9/merged major:0 minor:186 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/3b04d13addb01ef8ee9f4dbf9a7344ff3250bfc192f603793eb4c9d9320d6fa1/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-244:{mountpoint:/var/lib/containers/storage/overlay/e0807bd04186e821049be6d066a9ac9497762b84b590effbeb86219261a4748c/merged major:0 minor:244 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/70ec7d39235fabae8e5fe2f61dbcc1d26f39809c481aa87675083709d6c9a01e/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/7578bc08df3fcd270383addd684d302297118e3a7af11517b16dfc8f71dfd890/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/21aec52f5b5c9e6f3b64b9943331ed7cd5b8962bc633358cd04f497218319a8f/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/67987d8fc4a36870c885313e50e2c1bb0df4115897b2dbf66593f96c9d014b38/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/67f14836d1bb0772bde0fe6f8a680ffdd83fa0b87a6812073d102d400bbd2dc3/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/0cf80146111359e4bc4ae53af8c4b4fb8bc7b3643eda8e8fe2b9378ac123c417/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/a0fbf0b6b4237d6a2eb17486d69c5bd0df80a3c5d593539396bee8f20f1eb30a/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/96cfe444e08747582107cea9305df087f1c27bb35d01791f48d0095a28985569/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/c03c312ab044f8e30723b3c164fdbecebef1db9b2938c2434d18097c678bdd13/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/1d4a78b925673594ddae2e0be5491c0aaef821d75a469f937b23a119ccbfb3f1/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/69d6d2fa94e007b44b33b84e180d8be0ef3802d42ce35f76b33987a6d916240f/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/c97f2961a82977b9c7744edea1e6ef46505911bb4ee43c0b1cf7ff00d3c0aee2/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/8fc722d05558a18a01ff1a7a9bf712ace925fe2a74f0838a8d68f3ae01de630c/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/8bc0f8addca175115dc0897bcabbf55e428137a9f1b0328d784ffa9f10562307/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/bbd20cf49d230c59dbdc55d15026b62840a4ff9f86721138b13a4904ed876893/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/3d7f157b2819f4c8c7d739b4b1b2a39a17f3571d736f6f3d4e2300b949866eb6/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/d9f6b08e9e3465872482ea8c1b14e0e7f7b5d830e7e5723fea3a449cabbef499/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/669618383ca5ac49d15c862197e869aa72cd806d07c8bc845cb549f3074b6b5f/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/851f088888cf12d5d74965a5bc9b30f9d5612bcef45c4fdd00f17efe7cbdfbf5/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/a630fc11094cb2813166fd348a6c10908fefb6f929f45b0fb1670fb387911d21/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/f44cd0512c54597da2f75ca743539d327782780976ec5aa8082aa49c6698ed61/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/3210e0b5b96a0f85a803804c76f71421cf757f45e0e28a23014a101dc28d07cc/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/f43e77ed015f834f90fe09eb4c79fbf9695305d13a3d2f5ff08287836b615a96/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/a12fcca3e3ddf1edf57f1a554f3d63960f3fe7979bf6d38e341850d13bcb583f/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/5dbf2cac0e33550ee93e8562f0fdae6cf6ed4d305e6d6159d5ebf96235fb3b67/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/1b54ceac034e4364c818cbcc16bf2b780d899cfd1185abe12cb72fc92b54a074/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/ebe98d8c7ff4c2f56b01ca1d63a211dec9c9afd6d3bbcc6208a1a1cd39ebcc56/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/f778b614af1593910aaa07c314ed51f44c809b6c3030dd1242b7cd1322bfb25f/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/293a0744bc111eb00a417b1107b66d9bed152c87e3841a9af6ec38f6a6c8cbfe/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/db6197e983b25a262753075e4718e9f36f640299699e8f6ea900d6a368f660ad/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/649c73aede6493c9bc74d5fd36ac9b91e75a9d925ae01c783d5a3967143d44b8/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/2676abc72f851ed01ae96ecfc2d5bb9c477562b394c123f320984b664d1cd352/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/c1af558d397543d4e9fd6cf97bf502c997134b8f927e858b7799dfff528415fb/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/86aa36d9808ae303cb3e16d2a3dd0e14885aa80e42a9c0f8c1ecaf749a956491/merged major:0 minor:97 fsType:overlay blockSize:0}] Oct 30 23:06:20.914855 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.913309 4952 manager.go:217] Machine: {Timestamp:2026-10-30 23:06:20.912830136 +0000 UTC m=+0.056660332 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3128647 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cac8ecdb555cf4ae7cde59fb5c549 SystemUUID:ec2cac8e-cdb5-55cf-4ae7-cde59fb5c549 BootID:551ed6a1-ea6a-4c06-a2a8-3baec0ac93c0 Filesystems:[{Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b71dbcb555e71fae6aaffc5d8c19a7adc8dead3a2e5a786976813e1af918144a/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c9afe92d8ad57aa5fdeba3b2c942001aeb12544434f72c6727b33f2f0802d26d/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3838326a-857d-4511-8866-813015d85554/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9e5aeafe6cf21dbb40184cceedf15ae29eb2517a993918361607d6e3f5dde5fb/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4dd83f85253bebe12c121175d879cd3fd7e9e87a39ffe8f058ac37eee18f8c9c/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-186 DeviceMajor:0 DeviceMinor:186 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a7f82b8-115d-43ed-a7a2-cb869a7e48e0/volumes/kubernetes.io~projected/kube-api-access-rtk5p DeviceMajor:0 DeviceMinor:215 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/582d0663-6eba-4e69-81b0-fb25380f4c47/volumes/kubernetes.io~projected/kube-api-access-vg5n4 DeviceMajor:0 DeviceMinor:233 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a959ec9ff3ac805313fedb7b37a657941adadffaec8ffc81bace9e5ebc780d02/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c835c85a86c44cbbcd65500910f97f39ea2a458b5919bacf36b6eb4485c9ef4b/userdata/shm DeviceMajor:0 DeviceMinor:248 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bcddee82-a2d2-471b-badb-0295456e6a90/volumes/kubernetes.io~projected/kube-api-access-4b6bz DeviceMajor:0 DeviceMinor:329 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5f6c1ad7-10c4-4995-ae18-315d38184360/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f644db3c-e78a-42a8-9af3-c61194a21839/volumes/kubernetes.io~projected/kube-api-access-qbm49 DeviceMajor:0 DeviceMinor:218 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0e986cf43acdc91e04b55a1643afd28a907823e006cc7c87146f051e1aa17524/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/322e3a9b-911f-4cd8-8f43-71d56152e917/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:286 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d57a493fc14ecfecab1a187940f32a6090599edd683d5d078a4f75bb8e8846ad/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/724f369db6e1b26e89582a68d830fd9f8b170a494d63ea2253ef416049518117/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5cd7decf-5213-4360-a405-76d07c6ae3df/volumes/kubernetes.io~projected/kube-api-access-9tm5m DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e5deb16a572b4a313257ac5667bbc9a6f23ad2e38dfc6313dbf21bf25c868847/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/df5f40c01fa381a557e544f6888feca5bb2fe273560e70b3fa5669ae9a73ab02/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e32c5dc1-43ac-4ca5-a391-947b68d10d51/volumes/kubernetes.io~projected/kube-api-access-xp7ls DeviceMajor:0 DeviceMinor:224 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3838326a-857d-4511-8866-813015d85554/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ce66ace81ace169fbc0c516afacbdf9d1e4f43fb8bff281b5ae5dcfacf220b1d/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/edc2af65-1515-4742-89e1-b49814016b09/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:226 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/edc2af65-1515-4742-89e1-b49814016b09/volumes/kubernetes.io~projected/kube-api-access-6bxrl DeviceMajor:0 DeviceMinor:232 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4f7d22b5-256e-47d8-947f-b89b47f14d08/volumes/kubernetes.io~projected/kube-api-access-l9nfl DeviceMajor:0 DeviceMinor:287 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2861d402-9b72-4907-b8c1-7199ee3a8e92/volumes/kubernetes.io~projected/kube-api-access-pc5qz DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/706016da-3413-43e2-b06e-944572ca4106/volumes/kubernetes.io~projected/kube-api-access-7t9v8 DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e50f5624186bdf7c4bb7e28866f82676e6888120749c7b1056e4e72297fb3b6a/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/93ca6840cfd406c206a6093b26ff3ae4d0b6daef2ba2a11b345def032d54cd6f/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/volumes/kubernetes.io~projected/kube-api-access-b4cs6 DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5bfb1046-f8b5-4035-bb35-173a1316ea0f/volumes/kubernetes.io~projected/kube-api-access-lftl4 DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5f6c1ad7-10c4-4995-ae18-315d38184360/volumes/kubernetes.io~projected/kube-api-access-8czng DeviceMajor:0 DeviceMinor:213 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8cbb1813-35e1-49cb-8718-d0ba47d2350d/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9e0f56088f54a7f4479099ef4bd5f80a2b25ff5657f33c8920d01d9857369059/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3838326a-857d-4511-8866-813015d85554/volumes/kubernetes.io~projected/kube-api-access-rjr2r DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5/volumes/kubernetes.io~projected/kube-api-access-g48ld DeviceMajor:0 DeviceMinor:217 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-244 DeviceMajor:0 DeviceMinor:244 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/624d7fe2-4960-48ba-b711-3d85e4bf0687/volumes/kubernetes.io~projected/kube-api-access-gcls9 DeviceMajor:0 DeviceMinor:216 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8fe7a5b2e0ae165f7854fb568f1b3b75bc57b21564b5d529b3a2076811511adc/userdata/shm DeviceMajor:0 DeviceMinor:252 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e4d82728-b04b-4914-93d2-fce161acae36/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/67e30e6a-a537-4e2c-a484-f15281828fec/volumes/kubernetes.io~projected/kube-api-access-b754k DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3ec44853-19be-4bbb-9840-bb4c96fd2a29/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/58d18552-3df9-47f4-9291-b0e82cd954d2/volumes/kubernetes.io~projected/kube-api-access-zvlmc DeviceMajor:0 DeviceMinor:231 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/622a3003f71c337888542d7ca03cfeea3d5ba41630ac893ca1b000a06ddd87b9/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ec44853-19be-4bbb-9840-bb4c96fd2a29/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893/volumes/kubernetes.io~projected/kube-api-access-vvh4w DeviceMajor:0 DeviceMinor:222 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8cc1b5daf74f5223ffb2285e88662648c93d1d807a8cdc3775c9a4b57ec74333/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e9a89b14-a8e8-45f2-84bd-dba21617499c/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4374a26c15def0481cf21509e6531d6926bb66c03e0edd6e8e5b64fd66fc22e9/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8cbb1813-35e1-49cb-8718-d0ba47d2350d/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:223 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/582d0663-6eba-4e69-81b0-fb25380f4c47/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:225 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/487de86185e608fc2799e6118eb17391576d26d64e37d4d9ef3ae300dbe115d0/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8cbb1813-35e1-49cb-8718-d0ba47d2350d/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:205 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3ec44853-19be-4bbb-9840-bb4c96fd2a29/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:220 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c249bb339a7fff2e18422f1f388959232c4d30a100cdd65db5dfd96c0da2140b/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1cc88905-e18f-4a2d-a503-4b4950cd27d3/volumes/kubernetes.io~projected/kube-api-access-q42l2 DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a2e52979c7c424c66dfef4b2a3bb8bd8cb031109f65b823fb2725c331b66d839/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5f6c1ad7-10c4-4995-ae18-315d38184360/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e9a89b14-a8e8-45f2-84bd-dba21617499c/volumes/kubernetes.io~projected/kube-api-access-fc7nh DeviceMajor:0 DeviceMinor:212 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8cbb1813-35e1-49cb-8718-d0ba47d2350d/volumes/kubernetes.io~projected/kube-api-access-tpjxg DeviceMajor:0 DeviceMinor:214 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ef598d5f-a066-4b2d-96bd-fbfc47741dbb/volumes/kubernetes.io~projected/kube-api-access-kl67b DeviceMajor:0 DeviceMinor:219 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dd4fa69b3d38d1465c0b30f5f395947b0ff1d833244a71f20b8924d08ad75ed9/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dec2c5fa-4b9a-47ed-b005-04874761f419/volumes/kubernetes.io~projected/kube-api-access-r9g9l DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c4368706-761a-4f6e-9dda-a8087a5f4fdc/volumes/kubernetes.io~projected/kube-api-access-qtz5v DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3ec44853-19be-4bbb-9840-bb4c96fd2a29/volumes/kubernetes.io~projected/kube-api-access-pvnfz DeviceMajor:0 DeviceMinor:221 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/601f3571-aa13-43c6-8b5a-393029720fc3/volumes/kubernetes.io~projected/kube-api-access-rdsrp DeviceMajor:0 DeviceMinor:312 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef598d5f-a066-4b2d-96bd-fbfc47741dbb/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ad111ef6-4d18-4438-a0cc-e31a46804710/volumes/kubernetes.io~projected/kube-api-access-ldxjv DeviceMajor:0 DeviceMinor:211 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0e986cf43acdc91 MacAddress:02:bc:90:76:d9:b4 Speed:10000 Mtu:8901} {Name:4374a26c15def04 MacAddress:4a:c0:dd:fb:bd:6c Speed:10000 Mtu:8901} {Name:487de86185e608f MacAddress:6e:b1:ca:c3:54:e7 Speed:10000 Mtu:8901} {Name:622a3003f71c337 MacAddress:9a:61:8d:36:09:8b Speed:10000 Mtu:8901} {Name:724f369db6e1b26 MacAddress:8a:b4:7c:76:36:1d Speed:10000 Mtu:8901} {Name:8cc1b5daf74f522 MacAddress:a2:e9:d7:73:2d:6a Speed:10000 Mtu:8901} {Name:8fe7a5b2e0ae165 MacAddress:32:cb:03:48:61:c0 Speed:10000 Mtu:8901} {Name:9e0f56088f54a7f MacAddress:7a:60:d9:5d:e1:2a Speed:10000 Mtu:8901} {Name:9e5aeafe6cf21db MacAddress:0e:52:55:e8:c4:de Speed:10000 Mtu:8901} {Name:a959ec9ff3ac805 MacAddress:7a:19:9e:34:ca:07 Speed:10000 Mtu:8901} {Name:b71dbcb555e71fa MacAddress:e2:32:30:45:d4:01 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:02:3e:65:af:c9 Speed:0 Mtu:9001} {Name:br-int MacAddress:a2:80:45:fb:4a:44 Speed:0 Mtu:8901} {Name:c249bb339a7fff2 MacAddress:52:b6:d0:16:27:61 Speed:10000 Mtu:8901} {Name:c835c85a86c44cb MacAddress:1a:9e:2a:3e:44:f5 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:02:3e:65:af:c9 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:8a:de:49:be:08:15 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:4a:e2:3d:4e:d0:a7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 23:06:20.914855 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.913659 4952 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 23:06:20.914855 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.913699 4952 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 23:06:20.914855 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.914083 4952 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 23:06:20.914855 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.914101 4952 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-177.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 23:06:20.914855 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.914226 4952 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 23:06:20.914855 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.914234 4952 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 23:06:20.914855 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.914248 4952 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 23:06:20.914855 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.914261 4952 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 23:06:20.914855 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.914279 4952 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 23:06:20.914855 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.914475 4952 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 23:06:20.914855 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.914585 4952 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 23:06:20.914855 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.914622 4952 kubelet.go:493] "Attempting to sync node with API server" Oct 30 23:06:20.914855 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.914642 4952 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 23:06:20.914855 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.914651 4952 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 23:06:20.914855 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.914660 4952 kubelet.go:405] "Adding apiserver pod source" Oct 30 23:06:20.914855 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.914669 4952 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 23:06:20.915686 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.915628 4952 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 23:06:20.915721 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.915690 4952 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 23:06:20.916246 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.916222 4952 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 23:06:20.920449 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.920431 4952 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 23:06:20.920449 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.920447 4952 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 23:06:20.920583 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.920571 4952 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 23:06:20.920809 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.920795 4952 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 23:06:20.920848 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.920828 4952 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 23:06:20.920933 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.920924 4952 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 23:06:20.920962 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.920938 4952 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 23:06:20.920962 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.920944 4952 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 23:06:20.920962 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.920950 4952 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 23:06:20.920962 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.920956 4952 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 23:06:20.920962 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.920961 4952 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 23:06:20.921130 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.920967 4952 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 23:06:20.921130 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.920973 4952 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 23:06:20.921130 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.920980 4952 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 23:06:20.921130 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.920986 4952 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 23:06:20.921130 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.920994 4952 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 23:06:20.921130 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.921003 4952 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 23:06:20.921130 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.921029 4952 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 23:06:20.921130 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.921061 4952 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 23:06:20.924589 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.924576 4952 server.go:1263] "Started kubelet" Oct 30 23:06:20.924998 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.924661 4952 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 23:06:20.924998 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.924739 4952 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 23:06:20.924998 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.924816 4952 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 23:06:20.925210 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.925148 4952 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 23:06:20.925217 ip-10-0-130-177 systemd[1]: Started Kubernetes Kubelet. Oct 30 23:06:20.926105 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.926004 4952 server.go:317] "Adding debug handlers to kubelet server" Oct 30 23:06:20.929404 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.929387 4952 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 23:06:20.929586 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.929571 4952 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 23:06:20.929586 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.929587 4952 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 23:06:20.929712 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.929629 4952 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 23:06:20.929712 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.929662 4952 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 23:06:20.930051 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:20.929999 4952 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-177.us-west-1.compute.internal\" not found" Oct 30 23:06:20.931911 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.931687 4952 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:00:40 +0000 UTC" deadline="2028-05-18 15:43:01.527065179 +0000 UTC" Oct 30 23:06:20.931911 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.931729 4952 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13576h36m40.595340659s" Oct 30 23:06:20.932143 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.932128 4952 factory.go:55] Registering systemd factory Oct 30 23:06:20.932205 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.932148 4952 factory.go:223] Registration of the systemd container factory successfully Oct 30 23:06:20.933556 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.933537 4952 factory.go:153] Registering CRI-O factory Oct 30 23:06:20.933649 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.933563 4952 factory.go:223] Registration of the crio container factory successfully Oct 30 23:06:20.933649 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.933623 4952 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 23:06:20.933750 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.933641 4952 factory.go:103] Registering Raw factory Oct 30 23:06:20.933750 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.933670 4952 manager.go:1199] Started watching for new ooms in manager Oct 30 23:06:20.934087 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.934064 4952 manager.go:319] Starting recovery of all containers Oct 30 23:06:20.936462 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.936030 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec44853-19be-4bbb-9840-bb4c96fd2a29" volumeName="kubernetes.io/secret/3ec44853-19be-4bbb-9840-bb4c96fd2a29-image-registry-private-configuration" seLinuxMountContext="" Oct 30 23:06:20.936613 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.936471 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893" volumeName="kubernetes.io/secret/6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893-serving-cert" seLinuxMountContext="" Oct 30 23:06:20.936613 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.936508 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cbb1813-35e1-49cb-8718-d0ba47d2350d" volumeName="kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-bound-sa-token" seLinuxMountContext="" Oct 30 23:06:20.937125 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937103 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef598d5f-a066-4b2d-96bd-fbfc47741dbb" volumeName="kubernetes.io/projected/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-kube-api-access-kl67b" seLinuxMountContext="" Oct 30 23:06:20.937207 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937140 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="582d0663-6eba-4e69-81b0-fb25380f4c47" volumeName="kubernetes.io/secret/582d0663-6eba-4e69-81b0-fb25380f4c47-cert" seLinuxMountContext="" Oct 30 23:06:20.937207 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937156 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef598d5f-a066-4b2d-96bd-fbfc47741dbb" volumeName="kubernetes.io/empty-dir/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-tmp" seLinuxMountContext="" Oct 30 23:06:20.937207 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937171 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2861d402-9b72-4907-b8c1-7199ee3a8e92" volumeName="kubernetes.io/projected/2861d402-9b72-4907-b8c1-7199ee3a8e92-kube-api-access-pc5qz" seLinuxMountContext="" Oct 30 23:06:20.937207 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937189 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3838326a-857d-4511-8866-813015d85554" volumeName="kubernetes.io/empty-dir/3838326a-857d-4511-8866-813015d85554-etc-tuned" seLinuxMountContext="" Oct 30 23:06:20.937207 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937204 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bfb1046-f8b5-4035-bb35-173a1316ea0f" volumeName="kubernetes.io/configmap/5bfb1046-f8b5-4035-bb35-173a1316ea0f-multus-daemon-config" seLinuxMountContext="" Oct 30 23:06:20.937521 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937222 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="601f3571-aa13-43c6-8b5a-393029720fc3" volumeName="kubernetes.io/projected/601f3571-aa13-43c6-8b5a-393029720fc3-kube-api-access-rdsrp" seLinuxMountContext="" Oct 30 23:06:20.937521 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937235 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cbb1813-35e1-49cb-8718-d0ba47d2350d" volumeName="kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-kube-api-access-tpjxg" seLinuxMountContext="" Oct 30 23:06:20.937521 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937252 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cbb1813-35e1-49cb-8718-d0ba47d2350d" volumeName="kubernetes.io/secret/8cbb1813-35e1-49cb-8718-d0ba47d2350d-installation-pull-secrets" seLinuxMountContext="" Oct 30 23:06:20.937521 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937267 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef598d5f-a066-4b2d-96bd-fbfc47741dbb" volumeName="kubernetes.io/configmap/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-trusted-ca-bundle" seLinuxMountContext="" Oct 30 23:06:20.937521 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937279 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef598d5f-a066-4b2d-96bd-fbfc47741dbb" volumeName="kubernetes.io/empty-dir/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-snapshots" seLinuxMountContext="" Oct 30 23:06:20.937521 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937301 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3838326a-857d-4511-8866-813015d85554" volumeName="kubernetes.io/projected/3838326a-857d-4511-8866-813015d85554-kube-api-access-rjr2r" seLinuxMountContext="" Oct 30 23:06:20.937521 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937319 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec44853-19be-4bbb-9840-bb4c96fd2a29" volumeName="kubernetes.io/configmap/3ec44853-19be-4bbb-9840-bb4c96fd2a29-trusted-ca" seLinuxMountContext="" Oct 30 23:06:20.937521 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937347 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f644db3c-e78a-42a8-9af3-c61194a21839" volumeName="kubernetes.io/projected/f644db3c-e78a-42a8-9af3-c61194a21839-kube-api-access-qbm49" seLinuxMountContext="" Oct 30 23:06:20.937521 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937359 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20eb8eea-5fd8-42ab-bd78-8abc77f84eb2" volumeName="kubernetes.io/projected/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-kube-api-access-b4cs6" seLinuxMountContext="" Oct 30 23:06:20.937521 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937390 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="706016da-3413-43e2-b06e-944572ca4106" volumeName="kubernetes.io/configmap/706016da-3413-43e2-b06e-944572ca4106-iptables-alerter-script" seLinuxMountContext="" Oct 30 23:06:20.937521 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937402 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad111ef6-4d18-4438-a0cc-e31a46804710" volumeName="kubernetes.io/projected/ad111ef6-4d18-4438-a0cc-e31a46804710-kube-api-access-ldxjv" seLinuxMountContext="" Oct 30 23:06:20.937521 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937414 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcddee82-a2d2-471b-badb-0295456e6a90" volumeName="kubernetes.io/projected/bcddee82-a2d2-471b-badb-0295456e6a90-kube-api-access-4b6bz" seLinuxMountContext="" Oct 30 23:06:20.937521 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937430 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e32c5dc1-43ac-4ca5-a391-947b68d10d51" volumeName="kubernetes.io/projected/e32c5dc1-43ac-4ca5-a391-947b68d10d51-kube-api-access-xp7ls" seLinuxMountContext="" Oct 30 23:06:20.937521 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937441 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1cc88905-e18f-4a2d-a503-4b4950cd27d3" volumeName="kubernetes.io/projected/1cc88905-e18f-4a2d-a503-4b4950cd27d3-kube-api-access-q42l2" seLinuxMountContext="" Oct 30 23:06:20.937521 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937457 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58d18552-3df9-47f4-9291-b0e82cd954d2" volumeName="kubernetes.io/empty-dir/58d18552-3df9-47f4-9291-b0e82cd954d2-tmp-dir" seLinuxMountContext="" Oct 30 23:06:20.937521 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937468 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f6c1ad7-10c4-4995-ae18-315d38184360" volumeName="kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-stats-auth" seLinuxMountContext="" Oct 30 23:06:20.937521 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937479 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="624d7fe2-4960-48ba-b711-3d85e4bf0687" volumeName="kubernetes.io/projected/624d7fe2-4960-48ba-b711-3d85e4bf0687-kube-api-access-gcls9" seLinuxMountContext="" Oct 30 23:06:20.937521 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937495 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9a89b14-a8e8-45f2-84bd-dba21617499c" volumeName="kubernetes.io/configmap/e9a89b14-a8e8-45f2-84bd-dba21617499c-config" seLinuxMountContext="" Oct 30 23:06:20.937521 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937506 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="582d0663-6eba-4e69-81b0-fb25380f4c47" volumeName="kubernetes.io/projected/582d0663-6eba-4e69-81b0-fb25380f4c47-kube-api-access-vg5n4" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937535 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58d18552-3df9-47f4-9291-b0e82cd954d2" volumeName="kubernetes.io/configmap/58d18552-3df9-47f4-9291-b0e82cd954d2-config-volume" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937552 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bfb1046-f8b5-4035-bb35-173a1316ea0f" volumeName="kubernetes.io/projected/5bfb1046-f8b5-4035-bb35-173a1316ea0f-kube-api-access-lftl4" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937564 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f6c1ad7-10c4-4995-ae18-315d38184360" volumeName="kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-default-certificate" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937576 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cbb1813-35e1-49cb-8718-d0ba47d2350d" volumeName="kubernetes.io/configmap/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-certificates" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937592 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9a89b14-a8e8-45f2-84bd-dba21617499c" volumeName="kubernetes.io/projected/e9a89b14-a8e8-45f2-84bd-dba21617499c-kube-api-access-fc7nh" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937604 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec44853-19be-4bbb-9840-bb4c96fd2a29" volumeName="kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-kube-api-access-pvnfz" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937624 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cd7decf-5213-4360-a405-76d07c6ae3df" volumeName="kubernetes.io/empty-dir/5cd7decf-5213-4360-a405-76d07c6ae3df-tmp-dir" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937636 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893" volumeName="kubernetes.io/configmap/6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893-config" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937648 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cbb1813-35e1-49cb-8718-d0ba47d2350d" volumeName="kubernetes.io/configmap/8cbb1813-35e1-49cb-8718-d0ba47d2350d-trusted-ca" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937665 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20eb8eea-5fd8-42ab-bd78-8abc77f84eb2" volumeName="kubernetes.io/configmap/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-env-overrides" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937677 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58d18552-3df9-47f4-9291-b0e82cd954d2" volumeName="kubernetes.io/projected/58d18552-3df9-47f4-9291-b0e82cd954d2-kube-api-access-zvlmc" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937692 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893" volumeName="kubernetes.io/projected/6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893-kube-api-access-vvh4w" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937705 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4368706-761a-4f6e-9dda-a8087a5f4fdc" volumeName="kubernetes.io/configmap/c4368706-761a-4f6e-9dda-a8087a5f4fdc-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937716 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edc2af65-1515-4742-89e1-b49814016b09" volumeName="kubernetes.io/projected/edc2af65-1515-4742-89e1-b49814016b09-kube-api-access-6bxrl" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937733 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1cc88905-e18f-4a2d-a503-4b4950cd27d3" volumeName="kubernetes.io/configmap/1cc88905-e18f-4a2d-a503-4b4950cd27d3-serviceca" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937745 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="322e3a9b-911f-4cd8-8f43-71d56152e917" volumeName="kubernetes.io/secret/322e3a9b-911f-4cd8-8f43-71d56152e917-original-pull-secret" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937763 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67e30e6a-a537-4e2c-a484-f15281828fec" volumeName="kubernetes.io/projected/67e30e6a-a537-4e2c-a484-f15281828fec-kube-api-access-b754k" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937779 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef598d5f-a066-4b2d-96bd-fbfc47741dbb" volumeName="kubernetes.io/secret/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-serving-cert" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937791 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f7d22b5-256e-47d8-947f-b89b47f14d08" volumeName="kubernetes.io/projected/4f7d22b5-256e-47d8-947f-b89b47f14d08-kube-api-access-l9nfl" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937807 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5" volumeName="kubernetes.io/projected/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-kube-api-access-g48ld" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937818 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cbb1813-35e1-49cb-8718-d0ba47d2350d" volumeName="kubernetes.io/secret/8cbb1813-35e1-49cb-8718-d0ba47d2350d-image-registry-private-configuration" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937834 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9a89b14-a8e8-45f2-84bd-dba21617499c" volumeName="kubernetes.io/configmap/e9a89b14-a8e8-45f2-84bd-dba21617499c-trusted-ca" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937847 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9a89b14-a8e8-45f2-84bd-dba21617499c" volumeName="kubernetes.io/secret/e9a89b14-a8e8-45f2-84bd-dba21617499c-serving-cert" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937858 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0cb2a164-a8b5-4b93-b656-304550f04179" volumeName="kubernetes.io/configmap/0cb2a164-a8b5-4b93-b656-304550f04179-nginx-conf" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937874 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec44853-19be-4bbb-9840-bb4c96fd2a29" volumeName="kubernetes.io/secret/3ec44853-19be-4bbb-9840-bb4c96fd2a29-installation-pull-secrets" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937888 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="601f3571-aa13-43c6-8b5a-393029720fc3" volumeName="kubernetes.io/empty-dir/601f3571-aa13-43c6-8b5a-393029720fc3-data-volume" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937906 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5" volumeName="kubernetes.io/configmap/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-telemetry-config" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937918 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cbb1813-35e1-49cb-8718-d0ba47d2350d" volumeName="kubernetes.io/empty-dir/8cbb1813-35e1-49cb-8718-d0ba47d2350d-ca-trust-extracted" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937966 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f644db3c-e78a-42a8-9af3-c61194a21839" volumeName="kubernetes.io/configmap/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-config" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937983 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20eb8eea-5fd8-42ab-bd78-8abc77f84eb2" volumeName="kubernetes.io/secret/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.937996 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3838326a-857d-4511-8866-813015d85554" volumeName="kubernetes.io/empty-dir/3838326a-857d-4511-8866-813015d85554-tmp" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.938014 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cd7decf-5213-4360-a405-76d07c6ae3df" volumeName="kubernetes.io/projected/5cd7decf-5213-4360-a405-76d07c6ae3df-kube-api-access-9tm5m" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.938027 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4d82728-b04b-4914-93d2-fce161acae36" volumeName="kubernetes.io/secret/e4d82728-b04b-4914-93d2-fce161acae36-agent-certs" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.938097 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20eb8eea-5fd8-42ab-bd78-8abc77f84eb2" volumeName="kubernetes.io/configmap/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-ovnkube-config" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.938117 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec44853-19be-4bbb-9840-bb4c96fd2a29" volumeName="kubernetes.io/configmap/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-certificates" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.938129 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a7f82b8-115d-43ed-a7a2-cb869a7e48e0" volumeName="kubernetes.io/projected/8a7f82b8-115d-43ed-a7a2-cb869a7e48e0-kube-api-access-rtk5p" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.938145 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4d82728-b04b-4914-93d2-fce161acae36" volumeName="kubernetes.io/configmap/e4d82728-b04b-4914-93d2-fce161acae36-konnectivity-ca" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.938158 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bfb1046-f8b5-4035-bb35-173a1316ea0f" volumeName="kubernetes.io/configmap/5bfb1046-f8b5-4035-bb35-173a1316ea0f-cni-binary-copy" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.938174 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="706016da-3413-43e2-b06e-944572ca4106" volumeName="kubernetes.io/projected/706016da-3413-43e2-b06e-944572ca4106-kube-api-access-7t9v8" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.938193 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4368706-761a-4f6e-9dda-a8087a5f4fdc" volumeName="kubernetes.io/projected/c4368706-761a-4f6e-9dda-a8087a5f4fdc-kube-api-access-qtz5v" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.938206 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dec2c5fa-4b9a-47ed-b005-04874761f419" volumeName="kubernetes.io/projected/dec2c5fa-4b9a-47ed-b005-04874761f419-kube-api-access-r9g9l" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.938223 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec44853-19be-4bbb-9840-bb4c96fd2a29" volumeName="kubernetes.io/empty-dir/3ec44853-19be-4bbb-9840-bb4c96fd2a29-ca-trust-extracted" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.938236 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec44853-19be-4bbb-9840-bb4c96fd2a29" volumeName="kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-bound-sa-token" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.938251 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f6c1ad7-10c4-4995-ae18-315d38184360" volumeName="kubernetes.io/projected/5f6c1ad7-10c4-4995-ae18-315d38184360-kube-api-access-8czng" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.938273 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4368706-761a-4f6e-9dda-a8087a5f4fdc" volumeName="kubernetes.io/configmap/c4368706-761a-4f6e-9dda-a8087a5f4fdc-cni-binary-copy" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.938287 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4368706-761a-4f6e-9dda-a8087a5f4fdc" volumeName="kubernetes.io/configmap/c4368706-761a-4f6e-9dda-a8087a5f4fdc-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.938306 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edc2af65-1515-4742-89e1-b49814016b09" volumeName="kubernetes.io/configmap/edc2af65-1515-4742-89e1-b49814016b09-config" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.938318 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edc2af65-1515-4742-89e1-b49814016b09" volumeName="kubernetes.io/secret/edc2af65-1515-4742-89e1-b49814016b09-serving-cert" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.938332 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef598d5f-a066-4b2d-96bd-fbfc47741dbb" volumeName="kubernetes.io/configmap/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-service-ca-bundle" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.938349 4952 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20eb8eea-5fd8-42ab-bd78-8abc77f84eb2" volumeName="kubernetes.io/configmap/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-ovnkube-script-lib" seLinuxMountContext="" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.938361 4952 reconstruct.go:97] "Volume reconstruction finished" Oct 30 23:06:20.938408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.938369 4952 reconciler.go:29] "Reconciler: start to sync state" Oct 30 23:06:20.949705 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.949652 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/1.log" Oct 30 23:06:20.950231 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.950211 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/1.log" Oct 30 23:06:20.958249 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.958227 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 30 23:06:20.959253 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.959236 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 30 23:06:20.960958 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.960915 4952 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 23:06:20.962499 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.962479 4952 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 23:06:20.962578 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.962503 4952 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 23:06:20.962578 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.962527 4952 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 23:06:20.962681 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:20.962572 4952 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 23:06:20.974415 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.974394 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/1.log" Oct 30 23:06:20.974536 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.974420 4952 generic.go:356] "Generic (PLEG): container finished" podID="e9a89b14-a8e8-45f2-84bd-dba21617499c" containerID="066dcc6dcd6ce40d14f2a0ba4f1a61ca739e7a5dbee2bfa6413b3068b117427d" exitCode=255 Oct 30 23:06:20.985029 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.985001 4952 generic.go:356] "Generic (PLEG): container finished" podID="c4368706-761a-4f6e-9dda-a8087a5f4fdc" containerID="0e7f7a052841a0951ab3e18e295072dcb8e2f722fe318e4b586cdc60290545d6" exitCode=0 Oct 30 23:06:20.985029 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.985029 4952 generic.go:356] "Generic (PLEG): container finished" podID="c4368706-761a-4f6e-9dda-a8087a5f4fdc" containerID="be7f65ba4c1b84761a272cafb72344846bdc4c02c9e3931010e96d3e18a62792" exitCode=0 Oct 30 23:06:20.985180 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.985069 4952 generic.go:356] "Generic (PLEG): container finished" podID="c4368706-761a-4f6e-9dda-a8087a5f4fdc" containerID="77f2c64a49759466d37dfd95ca5ffe95365a543bb7af3853b857e74932b92e03" exitCode=0 Oct 30 23:06:20.985180 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.985078 4952 generic.go:356] "Generic (PLEG): container finished" podID="c4368706-761a-4f6e-9dda-a8087a5f4fdc" containerID="b495f09c9e4750943f4b3f475cb0a79ede3e9a2ed508965ba0958c5fbb8a6c4e" exitCode=0 Oct 30 23:06:20.985180 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.985087 4952 generic.go:356] "Generic (PLEG): container finished" podID="c4368706-761a-4f6e-9dda-a8087a5f4fdc" containerID="1f0300bb95e03d0a743ac258313f58bf64fe4f61739d233c8856bffbe41f47ad" exitCode=0 Oct 30 23:06:20.985180 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:20.985098 4952 generic.go:356] "Generic (PLEG): container finished" podID="c4368706-761a-4f6e-9dda-a8087a5f4fdc" containerID="6130681a4911f769eca5e7dd4bf58582e51e3c147984c1f714d5de9a4cdaa200" exitCode=0 Oct 30 23:06:21.002086 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.002066 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 30 23:06:21.002459 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.002436 4952 generic.go:356] "Generic (PLEG): container finished" podID="20eb8eea-5fd8-42ab-bd78-8abc77f84eb2" containerID="399eb8749765d7f02db13a1f1488b6b722b03162d5f0119e102148b1ad22f3b7" exitCode=1 Oct 30 23:06:21.004385 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.004363 4952 generic.go:356] "Generic (PLEG): container finished" podID="5d800e6e610855f53a5c54432ef51ba0" containerID="b1c73c5de5730069ed0ac6e21df2f41be2cd1dd3cc6448bd221f32fc113f1817" exitCode=0 Oct 30 23:06:21.029582 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.029566 4952 manager.go:324] Recovery completed Oct 30 23:06:21.030133 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:21.030117 4952 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-177.us-west-1.compute.internal\" not found" Oct 30 23:06:21.046899 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.046881 4952 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:06:21.048246 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.048224 4952 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:06:21.048326 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.048259 4952 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:06:21.048326 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.048274 4952 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 23:06:21.050770 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.050754 4952 cpu_manager.go:226] "Starting" policy="none" Oct 30 23:06:21.050770 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.050769 4952 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 23:06:21.050873 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.050770 4952 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:06:21.050873 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.050789 4952 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 23:06:21.050950 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.050938 4952 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 23:06:21.050981 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.050950 4952 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 23:06:21.050981 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.050967 4952 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 23:06:21.050981 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.050973 4952 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 23:06:21.050981 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.050978 4952 policy_none.go:50] "Start" Oct 30 23:06:21.051157 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.050986 4952 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 23:06:21.051157 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.050996 4952 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 23:06:21.051215 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.051181 4952 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 23:06:21.051215 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.051189 4952 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 23:06:21.051215 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.051195 4952 policy_none.go:44] "Start" Oct 30 23:06:21.051732 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.051715 4952 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:06:21.051791 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.051750 4952 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:06:21.051791 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.051767 4952 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 23:06:21.056977 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.056946 4952 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 23:06:21.059592 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.059577 4952 manager.go:341] "Starting Device Plugin manager" Oct 30 23:06:21.059663 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:21.059618 4952 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 23:06:21.059663 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.059631 4952 server.go:87] "Starting device plugin registration server" Oct 30 23:06:21.059949 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.059935 4952 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 23:06:21.059997 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.059950 4952 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 23:06:21.060081 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.060053 4952 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 23:06:21.060181 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.060146 4952 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 23:06:21.060246 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.060191 4952 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 23:06:21.060246 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.060199 4952 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 23:06:21.060332 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.060254 4952 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T23:06:21.060180074Z","UUID":"f83adc18-98ee-4053-b3db-e34f3c030a61","Handler":null,"Name":"","Endpoint":""} Oct 30 23:06:21.061435 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:21.061407 4952 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 23:06:21.061522 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:21.061445 4952 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-177.us-west-1.compute.internal\" not found" Oct 30 23:06:21.061622 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.061610 4952 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 23:06:21.061668 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.061629 4952 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 23:06:21.062656 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.062621 4952 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-177.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal"] Oct 30 23:06:21.062766 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.062752 4952 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:06:21.063757 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.063742 4952 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:06:21.063859 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.063772 4952 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:06:21.063859 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.063786 4952 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 23:06:21.064355 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.064134 4952 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:06:21.064355 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.064240 4952 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:06:21.065173 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.065141 4952 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:06:21.065173 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.065148 4952 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:06:21.065173 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.065170 4952 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:06:21.065173 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.065176 4952 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:06:21.065403 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.065185 4952 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 23:06:21.065403 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.065192 4952 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 23:06:21.065403 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:21.065325 4952 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-177.us-west-1.compute.internal\" not found" node="ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:06:21.065535 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.065505 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-177.us-west-1.compute.internal" event={"ID":"f343176fefb75130c5aad85365de861f","Type":"ContainerStarted","Data":"1dc97c46d12b832a1d1b2fb6e51032133d7255127ee94f24663d140ac07501e7"} Oct 30 23:06:21.065609 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.065546 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-177.us-west-1.compute.internal" event={"ID":"f343176fefb75130c5aad85365de861f","Type":"ContainerStarted","Data":"dd4fa69b3d38d1465c0b30f5f395947b0ff1d833244a71f20b8924d08ad75ed9"} Oct 30 23:06:21.065609 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.065587 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal" event={"ID":"5d800e6e610855f53a5c54432ef51ba0","Type":"ContainerStarted","Data":"06190e290792cc7dbc70297ed15f7f3f7656a69ade8ba2413405f9f1435da458"} Oct 30 23:06:21.065609 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.065604 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal" event={"ID":"5d800e6e610855f53a5c54432ef51ba0","Type":"ContainerDied","Data":"b1c73c5de5730069ed0ac6e21df2f41be2cd1dd3cc6448bd221f32fc113f1817"} Oct 30 23:06:21.065715 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.065623 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal" event={"ID":"5d800e6e610855f53a5c54432ef51ba0","Type":"ContainerStarted","Data":"d57a493fc14ecfecab1a187940f32a6090599edd683d5d078a4f75bb8e8846ad"} Oct 30 23:06:21.065715 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.065667 4952 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:06:21.066593 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.066578 4952 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:06:21.066680 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.066604 4952 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:06:21.066680 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.066619 4952 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 23:06:21.066786 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:21.066773 4952 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-177.us-west-1.compute.internal\" not found" node="ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:06:21.121230 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.121210 4952 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 23:06:21.121562 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.121547 4952 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:06:21.121860 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.121837 4952 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:06:21.121860 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.121845 4952 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:06:21.137172 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.137156 4952 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:06:21.160311 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.160294 4952 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:06:21.161314 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.161301 4952 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:06:21.161379 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.161327 4952 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:06:21.161379 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.161345 4952 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 23:06:21.161473 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.161383 4952 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:06:21.171427 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.171410 4952 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:06:21.212619 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.212572 4952 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:06:21.212619 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:21.212612 4952 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-177.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:06:21.212710 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.212623 4952 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:06:21.221455 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.221435 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5d800e6e610855f53a5c54432ef51ba0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal\" (UID: \"5d800e6e610855f53a5c54432ef51ba0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:06:21.221522 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.221459 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d800e6e610855f53a5c54432ef51ba0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal\" (UID: \"5d800e6e610855f53a5c54432ef51ba0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:06:21.221522 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.221479 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f343176fefb75130c5aad85365de861f-config\") pod \"kube-apiserver-proxy-ip-10-0-130-177.us-west-1.compute.internal\" (UID: \"f343176fefb75130c5aad85365de861f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:06:21.272501 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.272479 4952 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:06:21.272585 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.272538 4952 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:06:21.273009 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.272995 4952 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeNotReady" Oct 30 23:06:21.273111 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.273021 4952 setters.go:546] "Node became not ready" node="ip-10-0-130-177.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T23:06:21Z","lastTransitionTime":"2026-10-30T23:06:21Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Oct 30 23:06:21.298844 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.298818 4952 nodeinfomanager.go:471] existing CSINode "ip-10-0-130-177.us-west-1.compute.internal" is owned by different node (oldNodeID="daebfde3-a062-47ee-af1b-25122ce27f74", newNodeID=""), cleaning up... Oct 30 23:06:21.299934 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.299916 4952 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:06:21.300019 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:21.299954 4952 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:06:21.322090 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.322069 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f343176fefb75130c5aad85365de861f-config\") pod \"kube-apiserver-proxy-ip-10-0-130-177.us-west-1.compute.internal\" (UID: \"f343176fefb75130c5aad85365de861f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:06:21.322090 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.322082 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f343176fefb75130c5aad85365de861f-config\") pod \"kube-apiserver-proxy-ip-10-0-130-177.us-west-1.compute.internal\" (UID: \"f343176fefb75130c5aad85365de861f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:06:21.322216 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.322115 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5d800e6e610855f53a5c54432ef51ba0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal\" (UID: \"5d800e6e610855f53a5c54432ef51ba0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:06:21.322216 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.322137 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d800e6e610855f53a5c54432ef51ba0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal\" (UID: \"5d800e6e610855f53a5c54432ef51ba0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:06:21.322216 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.322143 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5d800e6e610855f53a5c54432ef51ba0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal\" (UID: \"5d800e6e610855f53a5c54432ef51ba0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:06:21.322216 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.322174 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d800e6e610855f53a5c54432ef51ba0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal\" (UID: \"5d800e6e610855f53a5c54432ef51ba0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:06:21.914969 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:21.914940 4952 apiserver.go:52] "Watching apiserver" Oct 30 23:06:22.020163 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.020139 4952 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:06:22.020325 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.020294 4952 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:06:22.067856 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.067830 4952 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 23:06:22.068603 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.068580 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-dh6nh","openshift-dns/dns-default-j9spz","kube-system/kas-connection-checker-685895c7c9-szm9n","openshift-service-ca-operator/service-ca-operator-544d749c8c-j7f9n","kube-system/kas-connection-checker-685895c7c9-lthcb","openshift-multus/multus-additional-cni-plugins-9cltq","openshift-console-operator/console-operator-7cd56b8dbc-kztwm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal","kube-system/global-pull-secret-syncer-7vrpr","openshift-insights/insights-operator-7c4986d449-g97cr","openshift-image-registry/image-registry-9c458c7df-njhmv","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5vcm","openshift-kube-storage-version-migrator/migrator-66df5d496b-xlp5h","openshift-network-diagnostics/network-check-target-xdlsv","openshift-network-console/networking-console-plugin-86c478bb88-r26cd","openshift-dns/node-resolver-p84ng","kube-system/kube-apiserver-proxy-ip-10-0-130-177.us-west-1.compute.internal","openshift-ingress/router-default-5dd6f58df-tctqk","kube-system/kas-connection-checker-685895c7c9-mpbb4","openshift-insights/insights-runtime-extractor-lfbzq","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chxn6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz","openshift-ovn-kubernetes/ovnkube-node-ts59n","openshift-image-registry/node-ca-74qh7","kube-system/konnectivity-agent-9sp4c","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kbrxt","openshift-multus/network-metrics-daemon-htqtp","openshift-network-operator/iptables-alerter-khr89","openshift-multus/multus-zx7t9","openshift-ingress-canary/ingress-canary-nzkrw","openshift-image-registry/image-registry-6f84d4ffd8-wx9nr","openshift-cluster-node-tuning-operator/tuned-z4l28","openshift-network-diagnostics/network-check-source-6f78d59f99-62rf9"] Oct 30 23:06:22.070296 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.070277 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:22.070867 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.070845 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dh6nh" Oct 30 23:06:22.071647 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.071630 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5vcm" Oct 30 23:06:22.072024 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.072003 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:22.073309 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.073293 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-r26cd" Oct 30 23:06:22.079200 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.079182 4952 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:06:22.079277 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.079226 4952 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-177.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:06:22.080634 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.080615 4952 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:06:22.080705 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.080656 4952 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-177.us-west-1.compute.internal" Oct 30 23:06:22.081763 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.081749 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:06:22.082254 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.082241 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:22.083270 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.083256 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j9spz" Oct 30 23:06:22.083752 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.083738 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:22.121735 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.121715 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 23:06:22.122152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.122139 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 23:06:22.123085 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.123070 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:06:22.123448 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.123434 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 23:06:22.123487 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.123438 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 23:06:22.123652 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.123639 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 23:06:22.123694 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.123664 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 23:06:22.123694 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.123666 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 23:06:22.124202 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.124191 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 23:06:22.124420 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.124410 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 23:06:22.124475 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.124465 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 23:06:22.124510 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.124501 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 23:06:22.125280 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.125267 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 23:06:22.125321 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.125283 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 23:06:22.125321 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.125305 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 23:06:22.125549 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.125537 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-2fmdc\"" Oct 30 23:06:22.132464 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.132450 4952 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 23:06:22.221991 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.221942 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 23:06:22.222293 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.222276 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 23:06:22.222293 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.222290 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 23:06:22.222392 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.222314 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 23:06:22.222392 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.222349 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 23:06:22.222392 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.222354 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 23:06:22.222392 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.222382 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 23:06:22.222551 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.222440 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 23:06:22.222810 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.222789 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 23:06:22.222867 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.222839 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 23:06:22.222867 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.222842 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 23:06:22.222867 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.222848 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 23:06:22.222990 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.222861 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 23:06:22.222990 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.222886 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 23:06:22.222990 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.222952 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 23:06:22.222990 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.222956 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 23:06:22.222990 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.222953 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 23:06:22.222990 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.222961 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 23:06:22.223291 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.223137 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 23:06:22.223291 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.223144 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:06:22.223291 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.223166 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 23:06:22.223291 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.223181 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 23:06:22.223291 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.223130 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:06:22.223291 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.223251 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 23:06:22.223538 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.223306 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 23:06:22.223538 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.223389 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:06:22.223538 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.223388 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:06:22.223538 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.223423 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 23:06:22.223538 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.223449 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 23:06:22.223707 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.223548 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:06:22.223707 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.223650 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 23:06:22.223707 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.223690 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 23:06:22.223846 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.223720 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 23:06:22.223846 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.223737 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 23:06:22.223846 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.223739 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 23:06:22.223846 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.223751 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 23:06:22.223846 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.223749 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 23:06:22.223846 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.223764 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 23:06:22.223846 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.223795 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 23:06:22.224067 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.223856 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 23:06:22.224067 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.223871 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 23:06:22.224067 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.223880 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:06:22.224067 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.223917 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 23:06:22.224067 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.223924 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 23:06:22.224067 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.223919 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 23:06:22.224067 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.223954 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 23:06:22.224067 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.223962 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 23:06:22.224067 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.223974 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 23:06:22.224067 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.223963 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 23:06:22.224067 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.224008 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 23:06:22.224067 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.224019 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 23:06:22.224397 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.224099 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 23:06:22.224397 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.224140 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 23:06:22.228518 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.228500 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8cbb1813-35e1-49cb-8718-d0ba47d2350d-trusted-ca\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:22.228564 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.228529 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-cni-netd\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.228564 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.228548 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-ovnkube-config\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.228639 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.228581 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-ovnkube-script-lib\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.228639 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.228626 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/601f3571-aa13-43c6-8b5a-393029720fc3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:22.228702 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.228695 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t9v8\" (UniqueName: \"kubernetes.io/projected/706016da-3413-43e2-b06e-944572ca4106-kube-api-access-7t9v8\") pod \"iptables-alerter-khr89\" (UID: \"706016da-3413-43e2-b06e-944572ca4106\") " pod="openshift-network-operator/iptables-alerter-khr89" Oct 30 23:06:22.228779 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.228759 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8czng\" (UniqueName: \"kubernetes.io/projected/5f6c1ad7-10c4-4995-ae18-315d38184360-kube-api-access-8czng\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:22.228812 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.228799 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-ovnkube-config\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.228867 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.228839 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-socket-dir\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:06:22.228908 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.228893 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-hostroot\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.228951 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.228932 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-slash\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.228982 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.228970 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bxrl\" (UniqueName: \"kubernetes.io/projected/edc2af65-1515-4742-89e1-b49814016b09-kube-api-access-6bxrl\") pod \"service-ca-operator-544d749c8c-j7f9n\" (UID: \"edc2af65-1515-4742-89e1-b49814016b09\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j7f9n" Oct 30 23:06:22.229018 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.229004 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbm49\" (UniqueName: \"kubernetes.io/projected/f644db3c-e78a-42a8-9af3-c61194a21839-kube-api-access-qbm49\") pod \"cluster-samples-operator-6ddb6c8b94-h5vcm\" (UID: \"f644db3c-e78a-42a8-9af3-c61194a21839\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5vcm" Oct 30 23:06:22.229113 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.229093 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-stats-auth\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:22.229573 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.229555 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-bound-sa-token\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:22.229616 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.229601 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-multus-conf-dir\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.229647 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.229635 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-service-ca-bundle\") pod \"insights-operator-7c4986d449-g97cr\" (UID: \"ef598d5f-a066-4b2d-96bd-fbfc47741dbb\") " pod="openshift-insights/insights-operator-7c4986d449-g97cr" Oct 30 23:06:22.229685 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.229651 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-run\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.229719 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.229695 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldxjv\" (UniqueName: \"kubernetes.io/projected/ad111ef6-4d18-4438-a0cc-e31a46804710-kube-api-access-ldxjv\") pod \"network-check-source-6f78d59f99-62rf9\" (UID: \"ad111ef6-4d18-4438-a0cc-e31a46804710\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-62rf9" Oct 30 23:06:22.229757 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.229739 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-stats-auth\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:22.229812 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.229787 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4368706-761a-4f6e-9dda-a8087a5f4fdc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:06:22.229849 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.229804 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-service-ca-bundle\") pod \"insights-operator-7c4986d449-g97cr\" (UID: \"ef598d5f-a066-4b2d-96bd-fbfc47741dbb\") " pod="openshift-insights/insights-operator-7c4986d449-g97cr" Oct 30 23:06:22.229849 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.229833 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-run-k8s-cni-cncf-io\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.229919 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.229884 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-var-lib-openvswitch\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.229919 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.229913 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcls9\" (UniqueName: \"kubernetes.io/projected/624d7fe2-4960-48ba-b711-3d85e4bf0687-kube-api-access-gcls9\") pod \"kas-connection-checker-685895c7c9-mpbb4\" (UID: \"624d7fe2-4960-48ba-b711-3d85e4bf0687\") " pod="kube-system/kas-connection-checker-685895c7c9-mpbb4" Oct 30 23:06:22.229975 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.229950 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-metrics-certs\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:22.230012 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.229975 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0cb2a164-a8b5-4b93-b656-304550f04179-nginx-conf\") pod \"networking-console-plugin-86c478bb88-r26cd\" (UID: \"0cb2a164-a8b5-4b93-b656-304550f04179\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r26cd" Oct 30 23:06:22.230065 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.230029 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5bfb1046-f8b5-4035-bb35-173a1316ea0f-multus-daemon-config\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.230116 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.230101 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-run-ovn-kubernetes\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.230190 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.230131 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/706016da-3413-43e2-b06e-944572ca4106-iptables-alerter-script\") pod \"iptables-alerter-khr89\" (UID: \"706016da-3413-43e2-b06e-944572ca4106\") " pod="openshift-network-operator/iptables-alerter-khr89" Oct 30 23:06:22.230190 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.230154 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-sysctl-d\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.230259 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.230201 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0cb2a164-a8b5-4b93-b656-304550f04179-nginx-conf\") pod \"networking-console-plugin-86c478bb88-r26cd\" (UID: \"0cb2a164-a8b5-4b93-b656-304550f04179\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r26cd" Oct 30 23:06:22.230259 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.230216 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtk5p\" (UniqueName: \"kubernetes.io/projected/8a7f82b8-115d-43ed-a7a2-cb869a7e48e0-kube-api-access-rtk5p\") pod \"kas-connection-checker-685895c7c9-lthcb\" (UID: \"8a7f82b8-115d-43ed-a7a2-cb869a7e48e0\") " pod="kube-system/kas-connection-checker-685895c7c9-lthcb" Oct 30 23:06:22.230259 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.230245 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h5vcm\" (UID: \"f644db3c-e78a-42a8-9af3-c61194a21839\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5vcm" Oct 30 23:06:22.230360 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.230273 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg5n4\" (UniqueName: \"kubernetes.io/projected/582d0663-6eba-4e69-81b0-fb25380f4c47-kube-api-access-vg5n4\") pod \"ingress-canary-nzkrw\" (UID: \"582d0663-6eba-4e69-81b0-fb25380f4c47\") " pod="openshift-ingress-canary/ingress-canary-nzkrw" Oct 30 23:06:22.230360 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.230297 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-device-dir\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:06:22.230360 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.230354 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e9a89b14-a8e8-45f2-84bd-dba21617499c-trusted-ca\") pod \"console-operator-7cd56b8dbc-kztwm\" (UID: \"e9a89b14-a8e8-45f2-84bd-dba21617499c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" Oct 30 23:06:22.230468 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.230367 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5bfb1046-f8b5-4035-bb35-173a1316ea0f-multus-daemon-config\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.230468 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.230385 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/322e3a9b-911f-4cd8-8f43-71d56152e917-original-pull-secret\") pod \"global-pull-secret-syncer-7vrpr\" (UID: \"322e3a9b-911f-4cd8-8f43-71d56152e917\") " pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:06:22.230468 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.230414 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/601f3571-aa13-43c6-8b5a-393029720fc3-data-volume\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:22.230468 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.230423 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/706016da-3413-43e2-b06e-944572ca4106-iptables-alerter-script\") pod \"iptables-alerter-khr89\" (UID: \"706016da-3413-43e2-b06e-944572ca4106\") " pod="openshift-network-operator/iptables-alerter-khr89" Oct 30 23:06:22.230468 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.230440 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp7ls\" (UniqueName: \"kubernetes.io/projected/e32c5dc1-43ac-4ca5-a391-947b68d10d51-kube-api-access-xp7ls\") pod \"volume-data-source-validator-58c9f5f46d-kbrxt\" (UID: \"e32c5dc1-43ac-4ca5-a391-947b68d10d51\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kbrxt" Oct 30 23:06:22.230468 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.230466 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tm5m\" (UniqueName: \"kubernetes.io/projected/5cd7decf-5213-4360-a405-76d07c6ae3df-kube-api-access-9tm5m\") pod \"node-resolver-p84ng\" (UID: \"5cd7decf-5213-4360-a405-76d07c6ae3df\") " pod="openshift-dns/node-resolver-p84ng" Oct 30 23:06:22.230706 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.230494 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bfb1046-f8b5-4035-bb35-173a1316ea0f-cni-binary-copy\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.230706 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.230513 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/601f3571-aa13-43c6-8b5a-393029720fc3-data-volume\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:22.230706 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.230604 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-node-log\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.230706 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.230629 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-etc-kubernetes\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.230706 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.230644 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-run-netns\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.230706 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.230661 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e9a89b14-a8e8-45f2-84bd-dba21617499c-serving-cert\") pod \"console-operator-7cd56b8dbc-kztwm\" (UID: \"e9a89b14-a8e8-45f2-84bd-dba21617499c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" Oct 30 23:06:22.230706 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.230667 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bfb1046-f8b5-4035-bb35-173a1316ea0f-cni-binary-copy\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.230997 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.230701 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/706016da-3413-43e2-b06e-944572ca4106-host-slash\") pod \"iptables-alerter-khr89\" (UID: \"706016da-3413-43e2-b06e-944572ca4106\") " pod="openshift-network-operator/iptables-alerter-khr89" Oct 30 23:06:22.230997 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.230750 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e9a89b14-a8e8-45f2-84bd-dba21617499c-config\") pod \"console-operator-7cd56b8dbc-kztwm\" (UID: \"e9a89b14-a8e8-45f2-84bd-dba21617499c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" Oct 30 23:06:22.230997 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.230766 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4368706-761a-4f6e-9dda-a8087a5f4fdc-cni-binary-copy\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:06:22.230997 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.230795 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e9a89b14-a8e8-45f2-84bd-dba21617499c-serving-cert\") pod \"console-operator-7cd56b8dbc-kztwm\" (UID: \"e9a89b14-a8e8-45f2-84bd-dba21617499c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" Oct 30 23:06:22.230997 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.230810 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b6bz\" (UniqueName: \"kubernetes.io/projected/bcddee82-a2d2-471b-badb-0295456e6a90-kube-api-access-4b6bz\") pod \"migrator-66df5d496b-xlp5h\" (UID: \"bcddee82-a2d2-471b-badb-0295456e6a90\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xlp5h" Oct 30 23:06:22.230997 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.230866 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58d18552-3df9-47f4-9291-b0e82cd954d2-config-volume\") pod \"dns-default-j9spz\" (UID: \"58d18552-3df9-47f4-9291-b0e82cd954d2\") " pod="openshift-dns/dns-default-j9spz" Oct 30 23:06:22.230997 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.230959 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-var-lib-cni-multus\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.230997 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.230979 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e9a89b14-a8e8-45f2-84bd-dba21617499c-config\") pod \"console-operator-7cd56b8dbc-kztwm\" (UID: \"e9a89b14-a8e8-45f2-84bd-dba21617499c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" Oct 30 23:06:22.230997 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.230979 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4368706-761a-4f6e-9dda-a8087a5f4fdc-cni-binary-copy\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:06:22.230997 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.230988 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f6c1ad7-10c4-4995-ae18-315d38184360-service-ca-bundle\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:22.231480 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231022 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-certificates\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:22.231480 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231069 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lftl4\" (UniqueName: \"kubernetes.io/projected/5bfb1046-f8b5-4035-bb35-173a1316ea0f-kube-api-access-lftl4\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.231480 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231094 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58d18552-3df9-47f4-9291-b0e82cd954d2-config-volume\") pod \"dns-default-j9spz\" (UID: \"58d18552-3df9-47f4-9291-b0e82cd954d2\") " pod="openshift-dns/dns-default-j9spz" Oct 30 23:06:22.231480 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231101 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-cni-bin\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.231480 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231132 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cd7decf-5213-4360-a405-76d07c6ae3df-tmp-dir\") pod \"node-resolver-p84ng\" (UID: \"5cd7decf-5213-4360-a405-76d07c6ae3df\") " pod="openshift-dns/node-resolver-p84ng" Oct 30 23:06:22.231480 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231145 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-certificates\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:22.231480 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231160 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-multus-cni-dir\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.231480 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231185 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-multus-socket-dir-parent\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.231480 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231208 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cd7decf-5213-4360-a405-76d07c6ae3df-tmp-dir\") pod \"node-resolver-p84ng\" (UID: \"5cd7decf-5213-4360-a405-76d07c6ae3df\") " pod="openshift-dns/node-resolver-p84ng" Oct 30 23:06:22.231480 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231227 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4cs6\" (UniqueName: \"kubernetes.io/projected/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-kube-api-access-b4cs6\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.231480 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231253 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ec44853-19be-4bbb-9840-bb4c96fd2a29-trusted-ca\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:22.231480 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231280 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-systemd\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.231480 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231307 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-certificates\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:22.231480 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231340 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-run-systemd\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.231480 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231374 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dh6nh\" (UID: \"813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dh6nh" Oct 30 23:06:22.231480 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231391 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58d18552-3df9-47f4-9291-b0e82cd954d2-tmp-dir\") pod \"dns-default-j9spz\" (UID: \"58d18552-3df9-47f4-9291-b0e82cd954d2\") " pod="openshift-dns/dns-default-j9spz" Oct 30 23:06:22.231480 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231407 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-system-cni-dir\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.231480 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231431 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-env-overrides\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.231480 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231455 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-certificates\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:22.231480 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231482 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58d18552-3df9-47f4-9291-b0e82cd954d2-tmp-dir\") pod \"dns-default-j9spz\" (UID: \"58d18552-3df9-47f4-9291-b0e82cd954d2\") " pod="openshift-dns/dns-default-j9spz" Oct 30 23:06:22.232093 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231527 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-serving-cert\") pod \"insights-operator-7c4986d449-g97cr\" (UID: \"ef598d5f-a066-4b2d-96bd-fbfc47741dbb\") " pod="openshift-insights/insights-operator-7c4986d449-g97cr" Oct 30 23:06:22.232093 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231546 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3838326a-857d-4511-8866-813015d85554-etc-tuned\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.232093 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231553 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-env-overrides\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.232093 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231564 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-kubelet\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.232093 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231613 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3838326a-857d-4511-8866-813015d85554-etc-tuned\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.232093 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231659 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-log-socket\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.232093 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231672 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-serving-cert\") pod \"insights-operator-7c4986d449-g97cr\" (UID: \"ef598d5f-a066-4b2d-96bd-fbfc47741dbb\") " pod="openshift-insights/insights-operator-7c4986d449-g97cr" Oct 30 23:06:22.232093 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231677 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.232093 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231717 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ec44853-19be-4bbb-9840-bb4c96fd2a29-image-registry-private-configuration\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:22.232093 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231758 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8cbb1813-35e1-49cb-8718-d0ba47d2350d-installation-pull-secrets\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:22.232093 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231831 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-registration-dir\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:06:22.232093 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231862 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-run-netns\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.232093 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231892 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-etc-openvswitch\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.232093 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231914 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-tls\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:22.232093 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231930 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-bound-sa-token\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:22.232093 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231945 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ec44853-19be-4bbb-9840-bb4c96fd2a29-image-registry-private-configuration\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:22.232093 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231945 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8cbb1813-35e1-49cb-8718-d0ba47d2350d-installation-pull-secrets\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:22.232093 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.231984 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q42l2\" (UniqueName: \"kubernetes.io/projected/1cc88905-e18f-4a2d-a503-4b4950cd27d3-kube-api-access-q42l2\") pod \"node-ca-74qh7\" (UID: \"1cc88905-e18f-4a2d-a503-4b4950cd27d3\") " pod="openshift-image-registry/node-ca-74qh7" Oct 30 23:06:22.232093 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232031 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjr2r\" (UniqueName: \"kubernetes.io/projected/3838326a-857d-4511-8866-813015d85554-kube-api-access-rjr2r\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.232841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232158 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-chxn6\" (UID: \"6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chxn6" Oct 30 23:06:22.232841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232195 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-default-certificate\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:22.232841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232227 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4368706-761a-4f6e-9dda-a8087a5f4fdc-system-cni-dir\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:06:22.232841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232291 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-run-ovn\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.232841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232308 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-chxn6\" (UID: \"6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chxn6" Oct 30 23:06:22.232841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232324 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/601f3571-aa13-43c6-8b5a-393029720fc3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:22.232841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232339 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-sysctl-conf\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.232841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232356 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvlmc\" (UniqueName: \"kubernetes.io/projected/58d18552-3df9-47f4-9291-b0e82cd954d2-kube-api-access-zvlmc\") pod \"dns-default-j9spz\" (UID: \"58d18552-3df9-47f4-9291-b0e82cd954d2\") " pod="openshift-dns/dns-default-j9spz" Oct 30 23:06:22.232841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232381 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1cc88905-e18f-4a2d-a503-4b4950cd27d3-serviceca\") pod \"node-ca-74qh7\" (UID: \"1cc88905-e18f-4a2d-a503-4b4950cd27d3\") " pod="openshift-image-registry/node-ca-74qh7" Oct 30 23:06:22.232841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232407 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-default-certificate\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:22.232841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232411 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-sys\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.232841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232429 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8cbb1813-35e1-49cb-8718-d0ba47d2350d-ca-trust-extracted\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:22.232841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232447 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc5qz\" (UniqueName: \"kubernetes.io/projected/2861d402-9b72-4907-b8c1-7199ee3a8e92-kube-api-access-pc5qz\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:06:22.232841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232462 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-run-multus-certs\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.232841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232494 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-run-openvswitch\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.232841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232518 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-snapshots\") pod \"insights-operator-7c4986d449-g97cr\" (UID: \"ef598d5f-a066-4b2d-96bd-fbfc47741dbb\") " pod="openshift-insights/insights-operator-7c4986d449-g97cr" Oct 30 23:06:22.232841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232542 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8cbb1813-35e1-49cb-8718-d0ba47d2350d-ca-trust-extracted\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:22.232841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232564 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs\") pod \"network-metrics-daemon-htqtp\" (UID: \"dec2c5fa-4b9a-47ed-b005-04874761f419\") " pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:06:22.232841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232581 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-snapshots\") pod \"insights-operator-7c4986d449-g97cr\" (UID: \"ef598d5f-a066-4b2d-96bd-fbfc47741dbb\") " pod="openshift-insights/insights-operator-7c4986d449-g97cr" Oct 30 23:06:22.232841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232572 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1cc88905-e18f-4a2d-a503-4b4950cd27d3-serviceca\") pod \"node-ca-74qh7\" (UID: \"1cc88905-e18f-4a2d-a503-4b4950cd27d3\") " pod="openshift-image-registry/node-ca-74qh7" Oct 30 23:06:22.232841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232603 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5cd7decf-5213-4360-a405-76d07c6ae3df-hosts-file\") pod \"node-resolver-p84ng\" (UID: \"5cd7decf-5213-4360-a405-76d07c6ae3df\") " pod="openshift-dns/node-resolver-p84ng" Oct 30 23:06:22.232841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232626 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e4d82728-b04b-4914-93d2-fce161acae36-agent-certs\") pod \"konnectivity-agent-9sp4c\" (UID: \"e4d82728-b04b-4914-93d2-fce161acae36\") " pod="kube-system/konnectivity-agent-9sp4c" Oct 30 23:06:22.232841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232647 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-host\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.232841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232687 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3838326a-857d-4511-8866-813015d85554-tmp\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.232841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232751 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-tls\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:22.232841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232768 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3838326a-857d-4511-8866-813015d85554-tmp\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.232841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232772 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl67b\" (UniqueName: \"kubernetes.io/projected/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-kube-api-access-kl67b\") pod \"insights-operator-7c4986d449-g97cr\" (UID: \"ef598d5f-a066-4b2d-96bd-fbfc47741dbb\") " pod="openshift-insights/insights-operator-7c4986d449-g97cr" Oct 30 23:06:22.232841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232805 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-lib-modules\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.232841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232832 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4368706-761a-4f6e-9dda-a8087a5f4fdc-cnibin\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:06:22.232841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232853 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c4368706-761a-4f6e-9dda-a8087a5f4fdc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232870 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc7nh\" (UniqueName: \"kubernetes.io/projected/e9a89b14-a8e8-45f2-84bd-dba21617499c-kube-api-access-fc7nh\") pod \"console-operator-7cd56b8dbc-kztwm\" (UID: \"e9a89b14-a8e8-45f2-84bd-dba21617499c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232886 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ec44853-19be-4bbb-9840-bb4c96fd2a29-ca-trust-extracted\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232902 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvnfz\" (UniqueName: \"kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-kube-api-access-pvnfz\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232919 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-kubernetes\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232946 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpjxg\" (UniqueName: \"kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-kube-api-access-tpjxg\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232970 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-cnibin\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.232977 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ec44853-19be-4bbb-9840-bb4c96fd2a29-ca-trust-extracted\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.233017 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdsrp\" (UniqueName: \"kubernetes.io/projected/601f3571-aa13-43c6-8b5a-393029720fc3-kube-api-access-rdsrp\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.233077 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c4368706-761a-4f6e-9dda-a8087a5f4fdc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.233119 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58d18552-3df9-47f4-9291-b0e82cd954d2-metrics-tls\") pod \"dns-default-j9spz\" (UID: \"58d18552-3df9-47f4-9291-b0e82cd954d2\") " pod="openshift-dns/dns-default-j9spz" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.233160 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/322e3a9b-911f-4cd8-8f43-71d56152e917-kubelet-config\") pod \"global-pull-secret-syncer-7vrpr\" (UID: \"322e3a9b-911f-4cd8-8f43-71d56152e917\") " pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.233208 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-modprobe-d\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.233257 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-h5vcm\" (UID: \"f644db3c-e78a-42a8-9af3-c61194a21839\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5vcm" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.233290 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c4368706-761a-4f6e-9dda-a8087a5f4fdc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.233318 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-etc-selinux\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.233345 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-ovn-node-metrics-cert\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.233375 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/edc2af65-1515-4742-89e1-b49814016b09-config\") pod \"service-ca-operator-544d749c8c-j7f9n\" (UID: \"edc2af65-1515-4742-89e1-b49814016b09\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j7f9n" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.233388 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-h5vcm\" (UID: \"f644db3c-e78a-42a8-9af3-c61194a21839\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5vcm" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.233401 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-sysconfig\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.233427 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-var-lib-kubelet\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.233460 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c4368706-761a-4f6e-9dda-a8087a5f4fdc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.233468 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b754k\" (UniqueName: \"kubernetes.io/projected/67e30e6a-a537-4e2c-a484-f15281828fec-kube-api-access-b754k\") pod \"kas-connection-checker-685895c7c9-szm9n\" (UID: \"67e30e6a-a537-4e2c-a484-f15281828fec\") " pod="kube-system/kas-connection-checker-685895c7c9-szm9n" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.233508 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9nfl\" (UniqueName: \"kubernetes.io/projected/4f7d22b5-256e-47d8-947f-b89b47f14d08-kube-api-access-l9nfl\") pod \"network-check-target-xdlsv\" (UID: \"4f7d22b5-256e-47d8-947f-b89b47f14d08\") " pod="openshift-network-diagnostics/network-check-target-xdlsv" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.233538 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/322e3a9b-911f-4cd8-8f43-71d56152e917-dbus\") pod \"global-pull-secret-syncer-7vrpr\" (UID: \"322e3a9b-911f-4cd8-8f43-71d56152e917\") " pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.233551 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/edc2af65-1515-4742-89e1-b49814016b09-config\") pod \"service-ca-operator-544d749c8c-j7f9n\" (UID: \"edc2af65-1515-4742-89e1-b49814016b09\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j7f9n" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.233568 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtz5v\" (UniqueName: \"kubernetes.io/projected/c4368706-761a-4f6e-9dda-a8087a5f4fdc-kube-api-access-qtz5v\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.233612 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0cb2a164-a8b5-4b93-b656-304550f04179-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-r26cd\" (UID: \"0cb2a164-a8b5-4b93-b656-304550f04179\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r26cd" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.233636 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ec44853-19be-4bbb-9840-bb4c96fd2a29-installation-pull-secrets\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.233651 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e4d82728-b04b-4914-93d2-fce161acae36-konnectivity-ca\") pod \"konnectivity-agent-9sp4c\" (UID: \"e4d82728-b04b-4914-93d2-fce161acae36\") " pod="kube-system/konnectivity-agent-9sp4c" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.233672 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/582d0663-6eba-4e69-81b0-fb25380f4c47-cert\") pod \"ingress-canary-nzkrw\" (UID: \"582d0663-6eba-4e69-81b0-fb25380f4c47\") " pod="openshift-ingress-canary/ingress-canary-nzkrw" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.233788 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8cbb1813-35e1-49cb-8718-d0ba47d2350d-image-registry-private-configuration\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.233809 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-dh6nh\" (UID: \"813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dh6nh" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.233825 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvh4w\" (UniqueName: \"kubernetes.io/projected/6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893-kube-api-access-vvh4w\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-chxn6\" (UID: \"6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chxn6" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.233849 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-var-lib-kubelet\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.233876 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g48ld\" (UniqueName: \"kubernetes.io/projected/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-kube-api-access-g48ld\") pod \"cluster-monitoring-operator-884576c57-dh6nh\" (UID: \"813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dh6nh" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.233882 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e4d82728-b04b-4914-93d2-fce161acae36-konnectivity-ca\") pod \"konnectivity-agent-9sp4c\" (UID: \"e4d82728-b04b-4914-93d2-fce161acae36\") " pod="kube-system/konnectivity-agent-9sp4c" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.233899 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9g9l\" (UniqueName: \"kubernetes.io/projected/dec2c5fa-4b9a-47ed-b005-04874761f419-kube-api-access-r9g9l\") pod \"network-metrics-daemon-htqtp\" (UID: \"dec2c5fa-4b9a-47ed-b005-04874761f419\") " pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.234015 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8cbb1813-35e1-49cb-8718-d0ba47d2350d-image-registry-private-configuration\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.234022 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/601f3571-aa13-43c6-8b5a-393029720fc3-crio-socket\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:22.234073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.234075 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-sys-fs\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:06:22.235239 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.234127 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-var-lib-cni-bin\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.235239 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.234183 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-g97cr\" (UID: \"ef598d5f-a066-4b2d-96bd-fbfc47741dbb\") " pod="openshift-insights/insights-operator-7c4986d449-g97cr" Oct 30 23:06:22.235239 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.234218 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/edc2af65-1515-4742-89e1-b49814016b09-serving-cert\") pod \"service-ca-operator-544d749c8c-j7f9n\" (UID: \"edc2af65-1515-4742-89e1-b49814016b09\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j7f9n" Oct 30 23:06:22.235239 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.234246 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1cc88905-e18f-4a2d-a503-4b4950cd27d3-host\") pod \"node-ca-74qh7\" (UID: \"1cc88905-e18f-4a2d-a503-4b4950cd27d3\") " pod="openshift-image-registry/node-ca-74qh7" Oct 30 23:06:22.235239 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.234286 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-chxn6\" (UID: \"6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chxn6" Oct 30 23:06:22.235239 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.234315 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-kubelet-dir\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:06:22.235239 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.234346 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-systemd-units\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.235239 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.234349 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/edc2af65-1515-4742-89e1-b49814016b09-serving-cert\") pod \"service-ca-operator-544d749c8c-j7f9n\" (UID: \"edc2af65-1515-4742-89e1-b49814016b09\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j7f9n" Oct 30 23:06:22.235239 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.234388 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ec44853-19be-4bbb-9840-bb4c96fd2a29-installation-pull-secrets\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:22.235239 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.234432 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-tmp\") pod \"insights-operator-7c4986d449-g97cr\" (UID: \"ef598d5f-a066-4b2d-96bd-fbfc47741dbb\") " pod="openshift-insights/insights-operator-7c4986d449-g97cr" Oct 30 23:06:22.235239 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.234542 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-tmp\") pod \"insights-operator-7c4986d449-g97cr\" (UID: \"ef598d5f-a066-4b2d-96bd-fbfc47741dbb\") " pod="openshift-insights/insights-operator-7c4986d449-g97cr" Oct 30 23:06:22.235239 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.234576 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-chxn6\" (UID: \"6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chxn6" Oct 30 23:06:22.277097 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.275362 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9xqsx\"" Oct 30 23:06:22.277097 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.275459 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7sr97\"" Oct 30 23:06:22.277097 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.275585 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-46zpf\"" Oct 30 23:06:22.277097 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.275666 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 23:06:22.277097 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.275712 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 23:06:22.277097 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.275925 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 23:06:22.277097 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.275985 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4j2qm\"" Oct 30 23:06:22.277097 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.276104 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fbzzd\"" Oct 30 23:06:22.277097 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.276304 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-7m4ss\"" Oct 30 23:06:22.277097 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.276368 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 23:06:22.277097 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.276390 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ldcvv\"" Oct 30 23:06:22.277097 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.276502 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t7twr\"" Oct 30 23:06:22.277097 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.276540 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 23:06:22.277097 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.276747 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nlg56\"" Oct 30 23:06:22.277756 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.277217 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 23:06:22.277756 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.277270 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 23:06:22.277756 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.277410 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 23:06:22.277756 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.277578 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 23:06:22.277756 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.277596 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-tw7s4\"" Oct 30 23:06:22.277978 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.277796 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 23:06:22.277978 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.277799 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 23:06:22.277978 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.277923 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 23:06:22.277978 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.277924 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-trr47\"" Oct 30 23:06:22.278539 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.278514 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-w9j6t\"" Oct 30 23:06:22.278539 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.278542 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cqcdm\"" Oct 30 23:06:22.278758 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.278526 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 23:06:22.278758 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.278584 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-l6bwx\"" Oct 30 23:06:22.278758 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.278710 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4qgbv\"" Oct 30 23:06:22.278758 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.278709 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-v482q\"" Oct 30 23:06:22.278987 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.278760 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cl4w9\"" Oct 30 23:06:22.279067 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.279012 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-ovnkube-script-lib\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.279067 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.279020 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 23:06:22.279139 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.279061 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8cbb1813-35e1-49cb-8718-d0ba47d2350d-trusted-ca\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:22.281656 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.281639 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/322e3a9b-911f-4cd8-8f43-71d56152e917-original-pull-secret\") pod \"global-pull-secret-syncer-7vrpr\" (UID: \"322e3a9b-911f-4cd8-8f43-71d56152e917\") " pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:06:22.281785 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.281764 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e9a89b14-a8e8-45f2-84bd-dba21617499c-trusted-ca\") pod \"console-operator-7cd56b8dbc-kztwm\" (UID: \"e9a89b14-a8e8-45f2-84bd-dba21617499c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" Oct 30 23:06:22.281823 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.281801 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ec44853-19be-4bbb-9840-bb4c96fd2a29-trusted-ca\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:22.282888 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.282873 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e4d82728-b04b-4914-93d2-fce161acae36-agent-certs\") pod \"konnectivity-agent-9sp4c\" (UID: \"e4d82728-b04b-4914-93d2-fce161acae36\") " pod="kube-system/konnectivity-agent-9sp4c" Oct 30 23:06:22.284001 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.283987 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/582d0663-6eba-4e69-81b0-fb25380f4c47-cert\") pod \"ingress-canary-nzkrw\" (UID: \"582d0663-6eba-4e69-81b0-fb25380f4c47\") " pod="openshift-ingress-canary/ingress-canary-nzkrw" Oct 30 23:06:22.284141 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.284127 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-ovn-node-metrics-cert\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.284187 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.284137 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-dh6nh\" (UID: \"813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dh6nh" Oct 30 23:06:22.284706 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.284689 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-g97cr\" (UID: \"ef598d5f-a066-4b2d-96bd-fbfc47741dbb\") " pod="openshift-insights/insights-operator-7c4986d449-g97cr" Oct 30 23:06:22.286072 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.286031 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bxrl\" (UniqueName: \"kubernetes.io/projected/edc2af65-1515-4742-89e1-b49814016b09-kube-api-access-6bxrl\") pod \"service-ca-operator-544d749c8c-j7f9n\" (UID: \"edc2af65-1515-4742-89e1-b49814016b09\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j7f9n" Oct 30 23:06:22.286201 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.286186 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8czng\" (UniqueName: \"kubernetes.io/projected/5f6c1ad7-10c4-4995-ae18-315d38184360-kube-api-access-8czng\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:22.286236 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.286203 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldxjv\" (UniqueName: \"kubernetes.io/projected/ad111ef6-4d18-4438-a0cc-e31a46804710-kube-api-access-ldxjv\") pod \"network-check-source-6f78d59f99-62rf9\" (UID: \"ad111ef6-4d18-4438-a0cc-e31a46804710\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-62rf9" Oct 30 23:06:22.288446 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.288430 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tm5m\" (UniqueName: \"kubernetes.io/projected/5cd7decf-5213-4360-a405-76d07c6ae3df-kube-api-access-9tm5m\") pod \"node-resolver-p84ng\" (UID: \"5cd7decf-5213-4360-a405-76d07c6ae3df\") " pod="openshift-dns/node-resolver-p84ng" Oct 30 23:06:22.291168 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.291150 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t9v8\" (UniqueName: \"kubernetes.io/projected/706016da-3413-43e2-b06e-944572ca4106-kube-api-access-7t9v8\") pod \"iptables-alerter-khr89\" (UID: \"706016da-3413-43e2-b06e-944572ca4106\") " pod="openshift-network-operator/iptables-alerter-khr89" Oct 30 23:06:22.292237 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.292219 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b6bz\" (UniqueName: \"kubernetes.io/projected/bcddee82-a2d2-471b-badb-0295456e6a90-kube-api-access-4b6bz\") pod \"migrator-66df5d496b-xlp5h\" (UID: \"bcddee82-a2d2-471b-badb-0295456e6a90\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xlp5h" Oct 30 23:06:22.292987 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.292750 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjr2r\" (UniqueName: \"kubernetes.io/projected/3838326a-857d-4511-8866-813015d85554-kube-api-access-rjr2r\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.292987 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.292819 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcls9\" (UniqueName: \"kubernetes.io/projected/624d7fe2-4960-48ba-b711-3d85e4bf0687-kube-api-access-gcls9\") pod \"kas-connection-checker-685895c7c9-mpbb4\" (UID: \"624d7fe2-4960-48ba-b711-3d85e4bf0687\") " pod="kube-system/kas-connection-checker-685895c7c9-mpbb4" Oct 30 23:06:22.292987 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.292852 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q42l2\" (UniqueName: \"kubernetes.io/projected/1cc88905-e18f-4a2d-a503-4b4950cd27d3-kube-api-access-q42l2\") pod \"node-ca-74qh7\" (UID: \"1cc88905-e18f-4a2d-a503-4b4950cd27d3\") " pod="openshift-image-registry/node-ca-74qh7" Oct 30 23:06:22.292987 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.292936 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-bound-sa-token\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:22.292987 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.292958 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lftl4\" (UniqueName: \"kubernetes.io/projected/5bfb1046-f8b5-4035-bb35-173a1316ea0f-kube-api-access-lftl4\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.293211 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.293072 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp7ls\" (UniqueName: \"kubernetes.io/projected/e32c5dc1-43ac-4ca5-a391-947b68d10d51-kube-api-access-xp7ls\") pod \"volume-data-source-validator-58c9f5f46d-kbrxt\" (UID: \"e32c5dc1-43ac-4ca5-a391-947b68d10d51\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-kbrxt" Oct 30 23:06:22.293211 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.293074 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg5n4\" (UniqueName: \"kubernetes.io/projected/582d0663-6eba-4e69-81b0-fb25380f4c47-kube-api-access-vg5n4\") pod \"ingress-canary-nzkrw\" (UID: \"582d0663-6eba-4e69-81b0-fb25380f4c47\") " pod="openshift-ingress-canary/ingress-canary-nzkrw" Oct 30 23:06:22.293470 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.293447 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtk5p\" (UniqueName: \"kubernetes.io/projected/8a7f82b8-115d-43ed-a7a2-cb869a7e48e0-kube-api-access-rtk5p\") pod \"kas-connection-checker-685895c7c9-lthcb\" (UID: \"8a7f82b8-115d-43ed-a7a2-cb869a7e48e0\") " pod="kube-system/kas-connection-checker-685895c7c9-lthcb" Oct 30 23:06:22.293567 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.293549 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbm49\" (UniqueName: \"kubernetes.io/projected/f644db3c-e78a-42a8-9af3-c61194a21839-kube-api-access-qbm49\") pod \"cluster-samples-operator-6ddb6c8b94-h5vcm\" (UID: \"f644db3c-e78a-42a8-9af3-c61194a21839\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5vcm" Oct 30 23:06:22.293659 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.293643 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc7nh\" (UniqueName: \"kubernetes.io/projected/e9a89b14-a8e8-45f2-84bd-dba21617499c-kube-api-access-fc7nh\") pod \"console-operator-7cd56b8dbc-kztwm\" (UID: \"e9a89b14-a8e8-45f2-84bd-dba21617499c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" Oct 30 23:06:22.293888 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.293858 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-bound-sa-token\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:22.294446 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.294427 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvh4w\" (UniqueName: \"kubernetes.io/projected/6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893-kube-api-access-vvh4w\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-chxn6\" (UID: \"6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chxn6" Oct 30 23:06:22.294565 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.294479 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4cs6\" (UniqueName: \"kubernetes.io/projected/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-kube-api-access-b4cs6\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.294565 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.294485 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g48ld\" (UniqueName: \"kubernetes.io/projected/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-kube-api-access-g48ld\") pod \"cluster-monitoring-operator-884576c57-dh6nh\" (UID: \"813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dh6nh" Oct 30 23:06:22.294943 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.294924 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvnfz\" (UniqueName: \"kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-kube-api-access-pvnfz\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:22.295318 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.295299 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtz5v\" (UniqueName: \"kubernetes.io/projected/c4368706-761a-4f6e-9dda-a8087a5f4fdc-kube-api-access-qtz5v\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:06:22.295384 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.295317 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b754k\" (UniqueName: \"kubernetes.io/projected/67e30e6a-a537-4e2c-a484-f15281828fec-kube-api-access-b754k\") pod \"kas-connection-checker-685895c7c9-szm9n\" (UID: \"67e30e6a-a537-4e2c-a484-f15281828fec\") " pod="kube-system/kas-connection-checker-685895c7c9-szm9n" Oct 30 23:06:22.295445 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.295435 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9nfl\" (UniqueName: \"kubernetes.io/projected/4f7d22b5-256e-47d8-947f-b89b47f14d08-kube-api-access-l9nfl\") pod \"network-check-target-xdlsv\" (UID: \"4f7d22b5-256e-47d8-947f-b89b47f14d08\") " pod="openshift-network-diagnostics/network-check-target-xdlsv" Oct 30 23:06:22.295888 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.295873 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvlmc\" (UniqueName: \"kubernetes.io/projected/58d18552-3df9-47f4-9291-b0e82cd954d2-kube-api-access-zvlmc\") pod \"dns-default-j9spz\" (UID: \"58d18552-3df9-47f4-9291-b0e82cd954d2\") " pod="openshift-dns/dns-default-j9spz" Oct 30 23:06:22.296330 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.296311 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9g9l\" (UniqueName: \"kubernetes.io/projected/dec2c5fa-4b9a-47ed-b005-04874761f419-kube-api-access-r9g9l\") pod \"network-metrics-daemon-htqtp\" (UID: \"dec2c5fa-4b9a-47ed-b005-04874761f419\") " pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:06:22.296402 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.296381 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl67b\" (UniqueName: \"kubernetes.io/projected/ef598d5f-a066-4b2d-96bd-fbfc47741dbb-kube-api-access-kl67b\") pod \"insights-operator-7c4986d449-g97cr\" (UID: \"ef598d5f-a066-4b2d-96bd-fbfc47741dbb\") " pod="openshift-insights/insights-operator-7c4986d449-g97cr" Oct 30 23:06:22.296701 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.296677 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpjxg\" (UniqueName: \"kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-kube-api-access-tpjxg\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:22.296766 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.296724 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdsrp\" (UniqueName: \"kubernetes.io/projected/601f3571-aa13-43c6-8b5a-393029720fc3-kube-api-access-rdsrp\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:22.296845 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.296830 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc5qz\" (UniqueName: \"kubernetes.io/projected/2861d402-9b72-4907-b8c1-7199ee3a8e92-kube-api-access-pc5qz\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:06:22.335391 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.335362 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0cb2a164-a8b5-4b93-b656-304550f04179-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-r26cd\" (UID: \"0cb2a164-a8b5-4b93-b656-304550f04179\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r26cd" Oct 30 23:06:22.335490 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.335397 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-var-lib-kubelet\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.335490 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.335421 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/601f3571-aa13-43c6-8b5a-393029720fc3-crio-socket\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:22.335490 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.335444 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-sys-fs\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:06:22.335490 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.335481 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-var-lib-cni-bin\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.335490 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.335485 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-var-lib-kubelet\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.335733 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.335503 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1cc88905-e18f-4a2d-a503-4b4950cd27d3-host\") pod \"node-ca-74qh7\" (UID: \"1cc88905-e18f-4a2d-a503-4b4950cd27d3\") " pod="openshift-image-registry/node-ca-74qh7" Oct 30 23:06:22.335733 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.335505 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/601f3571-aa13-43c6-8b5a-393029720fc3-crio-socket\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:22.335733 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.335509 4952 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 23:06:22.335733 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.335539 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-var-lib-cni-bin\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.335733 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.335543 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-sys-fs\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:06:22.335733 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.335552 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1cc88905-e18f-4a2d-a503-4b4950cd27d3-host\") pod \"node-ca-74qh7\" (UID: \"1cc88905-e18f-4a2d-a503-4b4950cd27d3\") " pod="openshift-image-registry/node-ca-74qh7" Oct 30 23:06:22.335733 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.335558 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-kubelet-dir\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:06:22.335733 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.335588 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-systemd-units\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.335733 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.335600 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-kubelet-dir\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:06:22.335733 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.335622 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cb2a164-a8b5-4b93-b656-304550f04179-networking-console-plugin-cert podName:0cb2a164-a8b5-4b93-b656-304550f04179 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:22.835589207 +0000 UTC m=+1.979419408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0cb2a164-a8b5-4b93-b656-304550f04179-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-r26cd" (UID: "0cb2a164-a8b5-4b93-b656-304550f04179") : secret "networking-console-plugin-cert" not found Oct 30 23:06:22.335733 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.335623 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-systemd-units\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.335733 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.335653 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-cni-netd\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.335733 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.335680 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/601f3571-aa13-43c6-8b5a-393029720fc3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:22.335733 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.335704 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-cni-netd\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.335733 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.335704 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-socket-dir\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:06:22.335733 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.335730 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-hostroot\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.335733 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.335744 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-slash\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.335753 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-socket-dir\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.335761 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-multus-conf-dir\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.335776 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-run\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.335794 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-hostroot\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.335834 4952 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.335860 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-multus-conf-dir\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.335871 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4368706-761a-4f6e-9dda-a8087a5f4fdc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.335896 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/601f3571-aa13-43c6-8b5a-393029720fc3-kube-rbac-proxy-cm podName:601f3571-aa13-43c6-8b5a-393029720fc3 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:22.835880247 +0000 UTC m=+1.979710446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/601f3571-aa13-43c6-8b5a-393029720fc3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lfbzq" (UID: "601f3571-aa13-43c6-8b5a-393029720fc3") : configmap "kube-rbac-proxy" not found Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.335899 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-slash\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.335902 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-run\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.335921 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-run-k8s-cni-cncf-io\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.335937 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-var-lib-openvswitch\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.335969 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-var-lib-openvswitch\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.335991 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-metrics-certs\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.335993 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4368706-761a-4f6e-9dda-a8087a5f4fdc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336017 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-run-k8s-cni-cncf-io\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336083 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-run-ovn-kubernetes\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.336091 4952 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336103 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-sysctl-d\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336121 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h5vcm\" (UID: \"f644db3c-e78a-42a8-9af3-c61194a21839\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5vcm" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.336140 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-metrics-certs podName:5f6c1ad7-10c4-4995-ae18-315d38184360 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:22.836129671 +0000 UTC m=+1.979959854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-metrics-certs") pod "router-default-5dd6f58df-tctqk" (UID: "5f6c1ad7-10c4-4995-ae18-315d38184360") : secret "router-metrics-certs-default" not found Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336151 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-device-dir\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336164 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-run-ovn-kubernetes\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336174 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-node-log\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.336188 4952 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336193 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-etc-kubernetes\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336196 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-device-dir\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336207 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-run-netns\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336223 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/706016da-3413-43e2-b06e-944572ca4106-host-slash\") pod \"iptables-alerter-khr89\" (UID: \"706016da-3413-43e2-b06e-944572ca4106\") " pod="openshift-network-operator/iptables-alerter-khr89" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336224 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-node-log\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.336236 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-tls podName:f644db3c-e78a-42a8-9af3-c61194a21839 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:22.836223925 +0000 UTC m=+1.980054110 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h5vcm" (UID: "f644db3c-e78a-42a8-9af3-c61194a21839") : secret "samples-operator-tls" not found Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336251 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-etc-kubernetes\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336259 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-run-netns\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336258 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-sysctl-d\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336268 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-var-lib-cni-multus\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336273 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/706016da-3413-43e2-b06e-944572ca4106-host-slash\") pod \"iptables-alerter-khr89\" (UID: \"706016da-3413-43e2-b06e-944572ca4106\") " pod="openshift-network-operator/iptables-alerter-khr89" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336291 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f6c1ad7-10c4-4995-ae18-315d38184360-service-ca-bundle\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336319 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-cni-bin\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336354 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-multus-cni-dir\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336356 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-var-lib-cni-multus\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336370 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-multus-socket-dir-parent\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336385 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-cni-bin\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336388 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-systemd\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.336401 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.336400 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f6c1ad7-10c4-4995-ae18-315d38184360-service-ca-bundle podName:5f6c1ad7-10c4-4995-ae18-315d38184360 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:22.836386563 +0000 UTC m=+1.980216752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5f6c1ad7-10c4-4995-ae18-315d38184360-service-ca-bundle") pod "router-default-5dd6f58df-tctqk" (UID: "5f6c1ad7-10c4-4995-ae18-315d38184360") : configmap references non-existent config key: service-ca.crt Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336437 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-systemd\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336440 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-multus-socket-dir-parent\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336438 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-run-systemd\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336473 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-run-systemd\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336442 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-multus-cni-dir\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336476 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dh6nh\" (UID: \"813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dh6nh" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.336539 4952 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336574 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-system-cni-dir\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336605 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-kubelet\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336622 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-log-socket\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336624 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-system-cni-dir\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.336631 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-cluster-monitoring-operator-tls podName:813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:22.83662045 +0000 UTC m=+1.980450633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-dh6nh" (UID: "813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5") : secret "cluster-monitoring-operator-tls" not found Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336642 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-log-socket\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336659 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336672 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-kubelet\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336678 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-registration-dir\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336704 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-run-netns\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336708 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-registration-dir\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336727 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336733 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-run-netns\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336729 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-etc-openvswitch\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336761 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-etc-openvswitch\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336763 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-tls\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336802 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4368706-761a-4f6e-9dda-a8087a5f4fdc-system-cni-dir\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.336812 4952 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.336822 4952 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9c458c7df-njhmv: secret "image-registry-tls" not found Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336827 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-run-ovn\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336853 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/601f3571-aa13-43c6-8b5a-393029720fc3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.336863 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-tls podName:3ec44853-19be-4bbb-9840-bb4c96fd2a29 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:22.83685252 +0000 UTC m=+1.980682744 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-tls") pod "image-registry-9c458c7df-njhmv" (UID: "3ec44853-19be-4bbb-9840-bb4c96fd2a29") : secret "image-registry-tls" not found Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336877 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4368706-761a-4f6e-9dda-a8087a5f4fdc-system-cni-dir\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336889 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-run-ovn\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336890 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-sysctl-conf\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.336922 4952 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336923 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-sys\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336949 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-run-multus-certs\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.336957 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/601f3571-aa13-43c6-8b5a-393029720fc3-insights-runtime-extractor-tls podName:601f3571-aa13-43c6-8b5a-393029720fc3 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:22.836948624 +0000 UTC m=+1.980778822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/601f3571-aa13-43c6-8b5a-393029720fc3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lfbzq" (UID: "601f3571-aa13-43c6-8b5a-393029720fc3") : secret "insights-runtime-extractor-tls" not found Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336972 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-sysctl-conf\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336974 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-run-openvswitch\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336983 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-host-run-multus-certs\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336992 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20eb8eea-5fd8-42ab-bd78-8abc77f84eb2-run-openvswitch\") pod \"ovnkube-node-ts59n\" (UID: \"20eb8eea-5fd8-42ab-bd78-8abc77f84eb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.336996 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs\") pod \"network-metrics-daemon-htqtp\" (UID: \"dec2c5fa-4b9a-47ed-b005-04874761f419\") " pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.337021 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5cd7decf-5213-4360-a405-76d07c6ae3df-hosts-file\") pod \"node-resolver-p84ng\" (UID: \"5cd7decf-5213-4360-a405-76d07c6ae3df\") " pod="openshift-dns/node-resolver-p84ng" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.337031 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-sys\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.337060 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-host\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.337104 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-host\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.337113 4952 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.337111 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-tls\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.337119 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5cd7decf-5213-4360-a405-76d07c6ae3df-hosts-file\") pod \"node-resolver-p84ng\" (UID: \"5cd7decf-5213-4360-a405-76d07c6ae3df\") " pod="openshift-dns/node-resolver-p84ng" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.337132 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-lib-modules\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.337152 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs podName:dec2c5fa-4b9a-47ed-b005-04874761f419 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:22.837141059 +0000 UTC m=+1.980971260 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs") pod "network-metrics-daemon-htqtp" (UID: "dec2c5fa-4b9a-47ed-b005-04874761f419") : secret "metrics-daemon-secret" not found Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.337165 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4368706-761a-4f6e-9dda-a8087a5f4fdc-cnibin\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.337172 4952 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.337185 4952 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f84d4ffd8-wx9nr: secret "image-registry-tls" not found Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.337186 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-kubernetes\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.337209 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-cnibin\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.337210 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-lib-modules\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.337220 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-tls podName:8cbb1813-35e1-49cb-8718-d0ba47d2350d nodeName:}" failed. No retries permitted until 2026-10-30 23:06:22.837207546 +0000 UTC m=+1.981037751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-tls") pod "image-registry-6f84d4ffd8-wx9nr" (UID: "8cbb1813-35e1-49cb-8718-d0ba47d2350d") : secret "image-registry-tls" not found Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.337230 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4368706-761a-4f6e-9dda-a8087a5f4fdc-cnibin\") pod \"multus-additional-cni-plugins-9cltq\" (UID: \"c4368706-761a-4f6e-9dda-a8087a5f4fdc\") " pod="openshift-multus/multus-additional-cni-plugins-9cltq" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.337245 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58d18552-3df9-47f4-9291-b0e82cd954d2-metrics-tls\") pod \"dns-default-j9spz\" (UID: \"58d18552-3df9-47f4-9291-b0e82cd954d2\") " pod="openshift-dns/dns-default-j9spz" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.337252 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-kubernetes\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.337260 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bfb1046-f8b5-4035-bb35-173a1316ea0f-cnibin\") pod \"multus-zx7t9\" (UID: \"5bfb1046-f8b5-4035-bb35-173a1316ea0f\") " pod="openshift-multus/multus-zx7t9" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.337288 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/322e3a9b-911f-4cd8-8f43-71d56152e917-kubelet-config\") pod \"global-pull-secret-syncer-7vrpr\" (UID: \"322e3a9b-911f-4cd8-8f43-71d56152e917\") " pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.337332 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-modprobe-d\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.337351 4952 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.337370 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-etc-selinux\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.337383 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58d18552-3df9-47f4-9291-b0e82cd954d2-metrics-tls podName:58d18552-3df9-47f4-9291-b0e82cd954d2 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:22.837373186 +0000 UTC m=+1.981203395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/58d18552-3df9-47f4-9291-b0e82cd954d2-metrics-tls") pod "dns-default-j9spz" (UID: "58d18552-3df9-47f4-9291-b0e82cd954d2") : secret "dns-default-metrics-tls" not found Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.337389 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/322e3a9b-911f-4cd8-8f43-71d56152e917-kubelet-config\") pod \"global-pull-secret-syncer-7vrpr\" (UID: \"322e3a9b-911f-4cd8-8f43-71d56152e917\") " pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.337413 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-sysconfig\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.337421 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2861d402-9b72-4907-b8c1-7199ee3a8e92-etc-selinux\") pod \"aws-ebs-csi-driver-node-df5rz\" (UID: \"2861d402-9b72-4907-b8c1-7199ee3a8e92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-df5rz" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.337429 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-var-lib-kubelet\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.337439 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-modprobe-d\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.337465 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-var-lib-kubelet\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.337470 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3838326a-857d-4511-8866-813015d85554-etc-sysconfig\") pod \"tuned-z4l28\" (UID: \"3838326a-857d-4511-8866-813015d85554\") " pod="openshift-cluster-node-tuning-operator/tuned-z4l28" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.337497 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/322e3a9b-911f-4cd8-8f43-71d56152e917-dbus\") pod \"global-pull-secret-syncer-7vrpr\" (UID: \"322e3a9b-911f-4cd8-8f43-71d56152e917\") " pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:06:22.338152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.337570 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/322e3a9b-911f-4cd8-8f43-71d56152e917-dbus\") pod \"global-pull-secret-syncer-7vrpr\" (UID: \"322e3a9b-911f-4cd8-8f43-71d56152e917\") " pod="kube-system/global-pull-secret-syncer-7vrpr" Oct 30 23:06:22.370356 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.370332 4952 scope.go:122] "RemoveContainer" containerID="066dcc6dcd6ce40d14f2a0ba4f1a61ca739e7a5dbee2bfa6413b3068b117427d" Oct 30 23:06:22.826128 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.826054 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:22.843323 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.843304 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f6c1ad7-10c4-4995-ae18-315d38184360-service-ca-bundle\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:22.843441 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.843337 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dh6nh\" (UID: \"813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dh6nh" Oct 30 23:06:22.843441 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.843362 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-tls\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:22.843441 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.843381 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/601f3571-aa13-43c6-8b5a-393029720fc3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:22.843441 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.843402 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs\") pod \"network-metrics-daemon-htqtp\" (UID: \"dec2c5fa-4b9a-47ed-b005-04874761f419\") " pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:06:22.843441 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.843428 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-tls\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:22.843653 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.843461 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58d18552-3df9-47f4-9291-b0e82cd954d2-metrics-tls\") pod \"dns-default-j9spz\" (UID: \"58d18552-3df9-47f4-9291-b0e82cd954d2\") " pod="openshift-dns/dns-default-j9spz" Oct 30 23:06:22.843653 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.843469 4952 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 23:06:22.843653 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.843482 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f6c1ad7-10c4-4995-ae18-315d38184360-service-ca-bundle podName:5f6c1ad7-10c4-4995-ae18-315d38184360 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:23.843460938 +0000 UTC m=+2.987291135 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5f6c1ad7-10c4-4995-ae18-315d38184360-service-ca-bundle") pod "router-default-5dd6f58df-tctqk" (UID: "5f6c1ad7-10c4-4995-ae18-315d38184360") : configmap references non-existent config key: service-ca.crt Oct 30 23:06:22.843653 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.843506 4952 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 23:06:22.843653 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.843512 4952 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 23:06:22.843653 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.843520 4952 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 23:06:22.843653 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.843527 4952 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:06:22.843653 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.843542 4952 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f84d4ffd8-wx9nr: secret "image-registry-tls" not found Oct 30 23:06:22.843653 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.843530 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-cluster-monitoring-operator-tls podName:813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:23.843514439 +0000 UTC m=+2.987344642 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-dh6nh" (UID: "813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5") : secret "cluster-monitoring-operator-tls" not found Oct 30 23:06:22.843653 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.843471 4952 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:06:22.843653 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.843586 4952 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9c458c7df-njhmv: secret "image-registry-tls" not found Oct 30 23:06:22.843653 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.843615 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs podName:dec2c5fa-4b9a-47ed-b005-04874761f419 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:23.843599621 +0000 UTC m=+2.987429803 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs") pod "network-metrics-daemon-htqtp" (UID: "dec2c5fa-4b9a-47ed-b005-04874761f419") : secret "metrics-daemon-secret" not found Oct 30 23:06:22.843653 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.843638 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/601f3571-aa13-43c6-8b5a-393029720fc3-insights-runtime-extractor-tls podName:601f3571-aa13-43c6-8b5a-393029720fc3 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:23.843627966 +0000 UTC m=+2.987458149 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/601f3571-aa13-43c6-8b5a-393029720fc3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lfbzq" (UID: "601f3571-aa13-43c6-8b5a-393029720fc3") : secret "insights-runtime-extractor-tls" not found Oct 30 23:06:22.843653 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.843649 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58d18552-3df9-47f4-9291-b0e82cd954d2-metrics-tls podName:58d18552-3df9-47f4-9291-b0e82cd954d2 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:23.843643336 +0000 UTC m=+2.987473519 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/58d18552-3df9-47f4-9291-b0e82cd954d2-metrics-tls") pod "dns-default-j9spz" (UID: "58d18552-3df9-47f4-9291-b0e82cd954d2") : secret "dns-default-metrics-tls" not found Oct 30 23:06:22.843653 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.843659 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-tls podName:8cbb1813-35e1-49cb-8718-d0ba47d2350d nodeName:}" failed. No retries permitted until 2026-10-30 23:06:23.843653778 +0000 UTC m=+2.987483961 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-tls") pod "image-registry-6f84d4ffd8-wx9nr" (UID: "8cbb1813-35e1-49cb-8718-d0ba47d2350d") : secret "image-registry-tls" not found Oct 30 23:06:22.844177 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.843680 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0cb2a164-a8b5-4b93-b656-304550f04179-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-r26cd\" (UID: \"0cb2a164-a8b5-4b93-b656-304550f04179\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r26cd" Oct 30 23:06:22.844177 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.843717 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/601f3571-aa13-43c6-8b5a-393029720fc3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:22.844177 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.843739 4952 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 23:06:22.844177 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.843745 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-metrics-certs\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:22.844177 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.843763 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-tls podName:3ec44853-19be-4bbb-9840-bb4c96fd2a29 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:23.843752371 +0000 UTC m=+2.987582555 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-tls") pod "image-registry-9c458c7df-njhmv" (UID: "3ec44853-19be-4bbb-9840-bb4c96fd2a29") : secret "image-registry-tls" not found Oct 30 23:06:22.844177 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.843788 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cb2a164-a8b5-4b93-b656-304550f04179-networking-console-plugin-cert podName:0cb2a164-a8b5-4b93-b656-304550f04179 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:23.84377203 +0000 UTC m=+2.987602232 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0cb2a164-a8b5-4b93-b656-304550f04179-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-r26cd" (UID: "0cb2a164-a8b5-4b93-b656-304550f04179") : secret "networking-console-plugin-cert" not found Oct 30 23:06:22.844177 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.843807 4952 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 23:06:22.844177 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.843824 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h5vcm\" (UID: \"f644db3c-e78a-42a8-9af3-c61194a21839\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5vcm" Oct 30 23:06:22.844177 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.843848 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-metrics-certs podName:5f6c1ad7-10c4-4995-ae18-315d38184360 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:23.843835481 +0000 UTC m=+2.987665663 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-metrics-certs") pod "router-default-5dd6f58df-tctqk" (UID: "5f6c1ad7-10c4-4995-ae18-315d38184360") : secret "router-metrics-certs-default" not found Oct 30 23:06:22.844177 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.843883 4952 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 23:06:22.844177 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:22.843913 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-tls podName:f644db3c-e78a-42a8-9af3-c61194a21839 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:23.843906029 +0000 UTC m=+2.987736212 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h5vcm" (UID: "f644db3c-e78a-42a8-9af3-c61194a21839") : secret "samples-operator-tls" not found Oct 30 23:06:22.844472 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:22.844217 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/601f3571-aa13-43c6-8b5a-393029720fc3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:23.029227 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:23.029201 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/2.log" Oct 30 23:06:23.029623 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:23.029608 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/1.log" Oct 30 23:06:23.029664 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:23.029642 4952 generic.go:356] "Generic (PLEG): container finished" podID="e9a89b14-a8e8-45f2-84bd-dba21617499c" containerID="ad8ae0b2f6a13762ab5345adf650e2cc6ec4532c73258014ffefcf23e32c4420" exitCode=255 Oct 30 23:06:23.029748 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:23.029725 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" event={"ID":"e9a89b14-a8e8-45f2-84bd-dba21617499c","Type":"ContainerDied","Data":"ad8ae0b2f6a13762ab5345adf650e2cc6ec4532c73258014ffefcf23e32c4420"} Oct 30 23:06:23.029826 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:23.029812 4952 scope.go:122] "RemoveContainer" containerID="066dcc6dcd6ce40d14f2a0ba4f1a61ca739e7a5dbee2bfa6413b3068b117427d" Oct 30 23:06:23.030220 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:23.030197 4952 scope.go:122] "RemoveContainer" containerID="ad8ae0b2f6a13762ab5345adf650e2cc6ec4532c73258014ffefcf23e32c4420" Oct 30 23:06:23.030386 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:23.030367 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-kztwm_openshift-console-operator(e9a89b14-a8e8-45f2-84bd-dba21617499c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" podUID="e9a89b14-a8e8-45f2-84bd-dba21617499c" Oct 30 23:06:23.070435 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:23.070413 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:23.198577 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:23.198545 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" Oct 30 23:06:23.852640 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:23.852609 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-metrics-certs\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:23.852640 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:23.852645 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h5vcm\" (UID: \"f644db3c-e78a-42a8-9af3-c61194a21839\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5vcm" Oct 30 23:06:23.852956 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:23.852672 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f6c1ad7-10c4-4995-ae18-315d38184360-service-ca-bundle\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:23.852956 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:23.852706 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dh6nh\" (UID: \"813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dh6nh" Oct 30 23:06:23.852956 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:23.852737 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-tls\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:23.852956 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:23.852766 4952 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 23:06:23.852956 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:23.852787 4952 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 23:06:23.852956 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:23.852766 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/601f3571-aa13-43c6-8b5a-393029720fc3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:23.852956 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:23.852808 4952 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 23:06:23.852956 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:23.852822 4952 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 23:06:23.852956 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:23.852814 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f6c1ad7-10c4-4995-ae18-315d38184360-service-ca-bundle podName:5f6c1ad7-10c4-4995-ae18-315d38184360 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:25.852795414 +0000 UTC m=+4.996625600 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5f6c1ad7-10c4-4995-ae18-315d38184360-service-ca-bundle") pod "router-default-5dd6f58df-tctqk" (UID: "5f6c1ad7-10c4-4995-ae18-315d38184360") : configmap references non-existent config key: service-ca.crt Oct 30 23:06:23.852956 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:23.852858 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-tls podName:f644db3c-e78a-42a8-9af3-c61194a21839 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:25.852845004 +0000 UTC m=+4.996675187 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h5vcm" (UID: "f644db3c-e78a-42a8-9af3-c61194a21839") : secret "samples-operator-tls" not found Oct 30 23:06:23.852956 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:23.852858 4952 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:06:23.852956 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:23.852874 4952 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9c458c7df-njhmv: secret "image-registry-tls" not found Oct 30 23:06:23.852956 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:23.852874 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-cluster-monitoring-operator-tls podName:813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:25.852864697 +0000 UTC m=+4.996694880 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-dh6nh" (UID: "813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5") : secret "cluster-monitoring-operator-tls" not found Oct 30 23:06:23.852956 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:23.852889 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-metrics-certs podName:5f6c1ad7-10c4-4995-ae18-315d38184360 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:25.852882813 +0000 UTC m=+4.996712996 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-metrics-certs") pod "router-default-5dd6f58df-tctqk" (UID: "5f6c1ad7-10c4-4995-ae18-315d38184360") : secret "router-metrics-certs-default" not found Oct 30 23:06:23.852956 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:23.852903 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/601f3571-aa13-43c6-8b5a-393029720fc3-insights-runtime-extractor-tls podName:601f3571-aa13-43c6-8b5a-393029720fc3 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:25.852895874 +0000 UTC m=+4.996726057 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/601f3571-aa13-43c6-8b5a-393029720fc3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lfbzq" (UID: "601f3571-aa13-43c6-8b5a-393029720fc3") : secret "insights-runtime-extractor-tls" not found Oct 30 23:06:23.852956 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:23.852921 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-tls podName:3ec44853-19be-4bbb-9840-bb4c96fd2a29 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:25.852915739 +0000 UTC m=+4.996745922 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-tls") pod "image-registry-9c458c7df-njhmv" (UID: "3ec44853-19be-4bbb-9840-bb4c96fd2a29") : secret "image-registry-tls" not found Oct 30 23:06:23.852956 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:23.852957 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs\") pod \"network-metrics-daemon-htqtp\" (UID: \"dec2c5fa-4b9a-47ed-b005-04874761f419\") " pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:06:23.853553 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:23.852977 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-tls\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:23.853553 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:23.853008 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58d18552-3df9-47f4-9291-b0e82cd954d2-metrics-tls\") pod \"dns-default-j9spz\" (UID: \"58d18552-3df9-47f4-9291-b0e82cd954d2\") " pod="openshift-dns/dns-default-j9spz" Oct 30 23:06:23.853553 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:23.853111 4952 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 23:06:23.853553 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:23.853122 4952 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 23:06:23.853553 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:23.853135 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58d18552-3df9-47f4-9291-b0e82cd954d2-metrics-tls podName:58d18552-3df9-47f4-9291-b0e82cd954d2 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:25.853128314 +0000 UTC m=+4.996958544 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/58d18552-3df9-47f4-9291-b0e82cd954d2-metrics-tls") pod "dns-default-j9spz" (UID: "58d18552-3df9-47f4-9291-b0e82cd954d2") : secret "dns-default-metrics-tls" not found Oct 30 23:06:23.853553 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:23.853137 4952 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f84d4ffd8-wx9nr: secret "image-registry-tls" not found Oct 30 23:06:23.853553 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:23.853158 4952 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 23:06:23.853553 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:23.853107 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0cb2a164-a8b5-4b93-b656-304550f04179-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-r26cd\" (UID: \"0cb2a164-a8b5-4b93-b656-304550f04179\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r26cd" Oct 30 23:06:23.853553 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:23.853174 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-tls podName:8cbb1813-35e1-49cb-8718-d0ba47d2350d nodeName:}" failed. No retries permitted until 2026-10-30 23:06:25.853160525 +0000 UTC m=+4.996990722 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-tls") pod "image-registry-6f84d4ffd8-wx9nr" (UID: "8cbb1813-35e1-49cb-8718-d0ba47d2350d") : secret "image-registry-tls" not found Oct 30 23:06:23.853553 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:23.853201 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cb2a164-a8b5-4b93-b656-304550f04179-networking-console-plugin-cert podName:0cb2a164-a8b5-4b93-b656-304550f04179 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:25.853194138 +0000 UTC m=+4.997024334 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0cb2a164-a8b5-4b93-b656-304550f04179-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-r26cd" (UID: "0cb2a164-a8b5-4b93-b656-304550f04179") : secret "networking-console-plugin-cert" not found Oct 30 23:06:23.853553 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:23.853161 4952 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 23:06:23.853553 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:23.853227 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs podName:dec2c5fa-4b9a-47ed-b005-04874761f419 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:25.853221405 +0000 UTC m=+4.997051633 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs") pod "network-metrics-daemon-htqtp" (UID: "dec2c5fa-4b9a-47ed-b005-04874761f419") : secret "metrics-daemon-secret" not found Oct 30 23:06:24.035115 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:24.035086 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/2.log" Oct 30 23:06:24.035520 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:24.035387 4952 scope.go:122] "RemoveContainer" containerID="ad8ae0b2f6a13762ab5345adf650e2cc6ec4532c73258014ffefcf23e32c4420" Oct 30 23:06:24.035572 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:24.035530 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-kztwm_openshift-console-operator(e9a89b14-a8e8-45f2-84bd-dba21617499c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" podUID="e9a89b14-a8e8-45f2-84bd-dba21617499c" Oct 30 23:06:24.162059 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:24.161965 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:24.557111 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:24.556983 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:24.746828 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:24.746804 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-p84ng_5cd7decf-5213-4360-a405-76d07c6ae3df/dns-node-resolver/0.log" Oct 30 23:06:24.769178 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:24.769154 4952 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" Oct 30 23:06:24.787835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:24.787811 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:25.018890 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:25.018851 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xdlsv" Oct 30 23:06:25.022646 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:25.022623 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:25.039769 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:25.039748 4952 scope.go:122] "RemoveContainer" containerID="ad8ae0b2f6a13762ab5345adf650e2cc6ec4532c73258014ffefcf23e32c4420" Oct 30 23:06:25.040475 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:25.040451 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-kztwm_openshift-console-operator(e9a89b14-a8e8-45f2-84bd-dba21617499c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" podUID="e9a89b14-a8e8-45f2-84bd-dba21617499c" Oct 30 23:06:25.241892 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:25.241862 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:25.468702 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:25.468669 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:25.731355 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:25.731280 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:25.746677 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:25.746658 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-74qh7_1cc88905-e18f-4a2d-a503-4b4950cd27d3/node-ca/0.log" Oct 30 23:06:25.869460 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:25.869436 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs\") pod \"network-metrics-daemon-htqtp\" (UID: \"dec2c5fa-4b9a-47ed-b005-04874761f419\") " pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:06:25.869581 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:25.869466 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-tls\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:25.869581 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:25.869491 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58d18552-3df9-47f4-9291-b0e82cd954d2-metrics-tls\") pod \"dns-default-j9spz\" (UID: \"58d18552-3df9-47f4-9291-b0e82cd954d2\") " pod="openshift-dns/dns-default-j9spz" Oct 30 23:06:25.869581 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:25.869516 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0cb2a164-a8b5-4b93-b656-304550f04179-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-r26cd\" (UID: \"0cb2a164-a8b5-4b93-b656-304550f04179\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r26cd" Oct 30 23:06:25.869679 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:25.869583 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-metrics-certs\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:25.869679 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:25.869601 4952 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 23:06:25.869679 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:25.869623 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h5vcm\" (UID: \"f644db3c-e78a-42a8-9af3-c61194a21839\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5vcm" Oct 30 23:06:25.869679 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:25.869655 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f6c1ad7-10c4-4995-ae18-315d38184360-service-ca-bundle\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:25.869679 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:25.869670 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs podName:dec2c5fa-4b9a-47ed-b005-04874761f419 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:29.869651875 +0000 UTC m=+9.013482069 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs") pod "network-metrics-daemon-htqtp" (UID: "dec2c5fa-4b9a-47ed-b005-04874761f419") : secret "metrics-daemon-secret" not found Oct 30 23:06:25.869903 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:25.869605 4952 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 23:06:25.869903 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:25.869715 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dh6nh\" (UID: \"813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dh6nh" Oct 30 23:06:25.869903 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:25.869727 4952 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 23:06:25.869903 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:25.869737 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f6c1ad7-10c4-4995-ae18-315d38184360-service-ca-bundle podName:5f6c1ad7-10c4-4995-ae18-315d38184360 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:29.869722932 +0000 UTC m=+9.013553135 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5f6c1ad7-10c4-4995-ae18-315d38184360-service-ca-bundle") pod "router-default-5dd6f58df-tctqk" (UID: "5f6c1ad7-10c4-4995-ae18-315d38184360") : configmap references non-existent config key: service-ca.crt Oct 30 23:06:25.869903 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:25.869730 4952 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 23:06:25.869903 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:25.869768 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cb2a164-a8b5-4b93-b656-304550f04179-networking-console-plugin-cert podName:0cb2a164-a8b5-4b93-b656-304550f04179 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:29.869750598 +0000 UTC m=+9.013580785 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0cb2a164-a8b5-4b93-b656-304550f04179-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-r26cd" (UID: "0cb2a164-a8b5-4b93-b656-304550f04179") : secret "networking-console-plugin-cert" not found Oct 30 23:06:25.869903 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:25.869780 4952 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 23:06:25.869903 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:25.869791 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58d18552-3df9-47f4-9291-b0e82cd954d2-metrics-tls podName:58d18552-3df9-47f4-9291-b0e82cd954d2 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:29.869781028 +0000 UTC m=+9.013611217 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/58d18552-3df9-47f4-9291-b0e82cd954d2-metrics-tls") pod "dns-default-j9spz" (UID: "58d18552-3df9-47f4-9291-b0e82cd954d2") : secret "dns-default-metrics-tls" not found Oct 30 23:06:25.869903 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:25.869808 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-metrics-certs podName:5f6c1ad7-10c4-4995-ae18-315d38184360 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:29.869800407 +0000 UTC m=+9.013630591 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-metrics-certs") pod "router-default-5dd6f58df-tctqk" (UID: "5f6c1ad7-10c4-4995-ae18-315d38184360") : secret "router-metrics-certs-default" not found Oct 30 23:06:25.869903 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:25.869834 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-cluster-monitoring-operator-tls podName:813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:29.86981978 +0000 UTC m=+9.013649964 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-dh6nh" (UID: "813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5") : secret "cluster-monitoring-operator-tls" not found Oct 30 23:06:25.869903 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:25.869884 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-tls\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:25.869903 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:25.869884 4952 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 23:06:25.870465 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:25.869917 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/601f3571-aa13-43c6-8b5a-393029720fc3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:25.870465 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:25.870014 4952 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 23:06:25.870465 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:25.870092 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/601f3571-aa13-43c6-8b5a-393029720fc3-insights-runtime-extractor-tls podName:601f3571-aa13-43c6-8b5a-393029720fc3 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:29.870067468 +0000 UTC m=+9.013897651 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/601f3571-aa13-43c6-8b5a-393029720fc3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lfbzq" (UID: "601f3571-aa13-43c6-8b5a-393029720fc3") : secret "insights-runtime-extractor-tls" not found Oct 30 23:06:25.870465 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:25.870212 4952 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 23:06:25.870465 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:25.870286 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-tls podName:f644db3c-e78a-42a8-9af3-c61194a21839 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:29.870269254 +0000 UTC m=+9.014099454 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h5vcm" (UID: "f644db3c-e78a-42a8-9af3-c61194a21839") : secret "samples-operator-tls" not found Oct 30 23:06:25.872243 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:25.872223 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-tls\") pod \"image-registry-6f84d4ffd8-wx9nr\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:25.872349 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:25.872333 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-tls\") pod \"image-registry-9c458c7df-njhmv\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:25.971160 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:25.971135 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:25.983472 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:25.983429 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:26.028402 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:26.027701 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:26.278989 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:26.278919 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:26.349894 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:26.349838 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nzkrw_582d0663-6eba-4e69-81b0-fb25380f4c47/serve-healthcheck-canary/0.log" Oct 30 23:06:26.444482 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:26.444448 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9c458c7df-njhmv"] Oct 30 23:06:26.446467 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:26.446446 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f84d4ffd8-wx9nr"] Oct 30 23:06:26.447399 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:26.447368 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec44853_19be_4bbb_9840_bb4c96fd2a29.slice/crio-b11590e240703cee89609a7a7f0225d852c3b0ebc0dbb695aa87035b33d995e2 WatchSource:0}: Error finding container b11590e240703cee89609a7a7f0225d852c3b0ebc0dbb695aa87035b33d995e2: Status 404 returned error can't find the container with id b11590e240703cee89609a7a7f0225d852c3b0ebc0dbb695aa87035b33d995e2 Oct 30 23:06:26.484961 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:26.484938 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cbb1813_35e1_49cb_8718_d0ba47d2350d.slice/crio-c33608cc1d1779d0316f484afb4803b78407e567af1f6ed38140bed3d7f29b74 WatchSource:0}: Error finding container c33608cc1d1779d0316f484afb4803b78407e567af1f6ed38140bed3d7f29b74: Status 404 returned error can't find the container with id c33608cc1d1779d0316f484afb4803b78407e567af1f6ed38140bed3d7f29b74 Oct 30 23:06:26.496357 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:26.496330 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:26.720737 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:26.720688 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:26.944639 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:26.944609 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:26.946078 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:26.946060 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-xlp5h_bcddee82-a2d2-471b-badb-0295456e6a90/migrator/0.log" Oct 30 23:06:27.047604 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:27.047541 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9c458c7df-njhmv" event={"ID":"3ec44853-19be-4bbb-9840-bb4c96fd2a29","Type":"ContainerStarted","Data":"fd91ccb839d39661c4e9fd1f00952839572ee83292febfc3d89fce1f77ab9524"} Oct 30 23:06:27.047604 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:27.047592 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9c458c7df-njhmv" event={"ID":"3ec44853-19be-4bbb-9840-bb4c96fd2a29","Type":"ContainerStarted","Data":"b11590e240703cee89609a7a7f0225d852c3b0ebc0dbb695aa87035b33d995e2"} Oct 30 23:06:27.049079 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:27.049049 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" event={"ID":"8cbb1813-35e1-49cb-8718-d0ba47d2350d","Type":"ContainerStarted","Data":"1fa807a31856c6c15abfcb704388b03330af1626dd7ea009cf1fb5b2bbcb5a0b"} Oct 30 23:06:27.049154 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:27.049094 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" event={"ID":"8cbb1813-35e1-49cb-8718-d0ba47d2350d","Type":"ContainerStarted","Data":"c33608cc1d1779d0316f484afb4803b78407e567af1f6ed38140bed3d7f29b74"} Oct 30 23:06:27.148221 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:27.148188 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-chxn6_6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893/kube-storage-version-migrator-operator/0.log" Oct 30 23:06:27.327378 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:27.327320 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:27.375291 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:27.375265 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:27.958242 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:27.958213 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:28.033143 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:28.033111 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:28.052633 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:28.052610 4952 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 23:06:28.052753 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:28.052718 4952 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 23:06:28.270497 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:28.270430 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:28.489877 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:28.489849 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:28.715245 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:28.715217 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:28.724721 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:28.724701 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ts59n" Oct 30 23:06:28.923720 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:28.923693 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:29.907196 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:29.907164 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f6c1ad7-10c4-4995-ae18-315d38184360-service-ca-bundle\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:29.907196 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:29.907204 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dh6nh\" (UID: \"813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dh6nh" Oct 30 23:06:29.907725 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:29.907230 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/601f3571-aa13-43c6-8b5a-393029720fc3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:29.907725 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:29.907255 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs\") pod \"network-metrics-daemon-htqtp\" (UID: \"dec2c5fa-4b9a-47ed-b005-04874761f419\") " pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:06:29.907725 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:29.907289 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58d18552-3df9-47f4-9291-b0e82cd954d2-metrics-tls\") pod \"dns-default-j9spz\" (UID: \"58d18552-3df9-47f4-9291-b0e82cd954d2\") " pod="openshift-dns/dns-default-j9spz" Oct 30 23:06:29.907725 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:29.907323 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0cb2a164-a8b5-4b93-b656-304550f04179-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-r26cd\" (UID: \"0cb2a164-a8b5-4b93-b656-304550f04179\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r26cd" Oct 30 23:06:29.907725 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:29.907328 4952 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 23:06:29.907725 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:29.907361 4952 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 23:06:29.907725 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:29.907374 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-metrics-certs\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:29.907725 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:29.907405 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h5vcm\" (UID: \"f644db3c-e78a-42a8-9af3-c61194a21839\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5vcm" Oct 30 23:06:29.907725 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:29.907407 4952 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 23:06:29.907725 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:29.907437 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/601f3571-aa13-43c6-8b5a-393029720fc3-insights-runtime-extractor-tls podName:601f3571-aa13-43c6-8b5a-393029720fc3 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:37.907417707 +0000 UTC m=+17.051247908 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/601f3571-aa13-43c6-8b5a-393029720fc3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lfbzq" (UID: "601f3571-aa13-43c6-8b5a-393029720fc3") : secret "insights-runtime-extractor-tls" not found Oct 30 23:06:29.907725 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:29.907461 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58d18552-3df9-47f4-9291-b0e82cd954d2-metrics-tls podName:58d18552-3df9-47f4-9291-b0e82cd954d2 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:37.907450868 +0000 UTC m=+17.051281053 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/58d18552-3df9-47f4-9291-b0e82cd954d2-metrics-tls") pod "dns-default-j9spz" (UID: "58d18552-3df9-47f4-9291-b0e82cd954d2") : secret "dns-default-metrics-tls" not found Oct 30 23:06:29.907725 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:29.907501 4952 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 23:06:29.907725 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:29.907517 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-cluster-monitoring-operator-tls podName:813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:37.907507548 +0000 UTC m=+17.051337736 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-dh6nh" (UID: "813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5") : secret "cluster-monitoring-operator-tls" not found Oct 30 23:06:29.907725 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:29.907584 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cb2a164-a8b5-4b93-b656-304550f04179-networking-console-plugin-cert podName:0cb2a164-a8b5-4b93-b656-304550f04179 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:37.907566812 +0000 UTC m=+17.051396994 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0cb2a164-a8b5-4b93-b656-304550f04179-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-r26cd" (UID: "0cb2a164-a8b5-4b93-b656-304550f04179") : secret "networking-console-plugin-cert" not found Oct 30 23:06:29.907725 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:29.907506 4952 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 23:06:29.907725 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:29.907625 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-tls podName:f644db3c-e78a-42a8-9af3-c61194a21839 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:37.907616762 +0000 UTC m=+17.051446958 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h5vcm" (UID: "f644db3c-e78a-42a8-9af3-c61194a21839") : secret "samples-operator-tls" not found Oct 30 23:06:29.908272 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:29.907915 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f6c1ad7-10c4-4995-ae18-315d38184360-service-ca-bundle\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:29.909834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:29.909810 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dec2c5fa-4b9a-47ed-b005-04874761f419-metrics-certs\") pod \"network-metrics-daemon-htqtp\" (UID: \"dec2c5fa-4b9a-47ed-b005-04874761f419\") " pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:06:29.909935 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:29.909860 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6c1ad7-10c4-4995-ae18-315d38184360-metrics-certs\") pod \"router-default-5dd6f58df-tctqk\" (UID: \"5f6c1ad7-10c4-4995-ae18-315d38184360\") " pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:30.013971 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:30.013945 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:30.173194 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:30.173129 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:30.182863 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:30.182833 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-htqtp" Oct 30 23:06:30.225878 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:30.225766 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:30.435395 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:30.435313 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:30.641556 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:30.641529 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-htqtp"] Oct 30 23:06:30.644812 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:30.644783 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddec2c5fa_4b9a_47ed_b005_04874761f419.slice/crio-7fd2e16a125a37ec459726d965b0bf59fca66cc39c0d73e2ff2d16de2968c50c WatchSource:0}: Error finding container 7fd2e16a125a37ec459726d965b0bf59fca66cc39c0d73e2ff2d16de2968c50c: Status 404 returned error can't find the container with id 7fd2e16a125a37ec459726d965b0bf59fca66cc39c0d73e2ff2d16de2968c50c Oct 30 23:06:30.645026 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:30.645009 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5dd6f58df-tctqk"] Oct 30 23:06:30.646776 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:30.646763 4952 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 23:06:30.655702 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:30.655682 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:30.695188 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:30.695168 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f6c1ad7_10c4_4995_ae18_315d38184360.slice/crio-3cb79cdccbe3d4c08487d9b9b854bd16a4114b0fc0b1ba4601828ba6155b4fd0 WatchSource:0}: Error finding container 3cb79cdccbe3d4c08487d9b9b854bd16a4114b0fc0b1ba4601828ba6155b4fd0: Status 404 returned error can't find the container with id 3cb79cdccbe3d4c08487d9b9b854bd16a4114b0fc0b1ba4601828ba6155b4fd0 Oct 30 23:06:30.868052 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:30.868008 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:31.063908 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:31.063825 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-htqtp" event={"ID":"dec2c5fa-4b9a-47ed-b005-04874761f419","Type":"ContainerStarted","Data":"7fd2e16a125a37ec459726d965b0bf59fca66cc39c0d73e2ff2d16de2968c50c"} Oct 30 23:06:31.065553 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:31.065523 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5dd6f58df-tctqk" event={"ID":"5f6c1ad7-10c4-4995-ae18-315d38184360","Type":"ContainerStarted","Data":"568d933ca6edb988d8de09cacf56f06dc4db2bac1795c21e3a91b90a0866a8c3"} Oct 30 23:06:31.065689 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:31.065564 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5dd6f58df-tctqk" event={"ID":"5f6c1ad7-10c4-4995-ae18-315d38184360","Type":"ContainerStarted","Data":"3cb79cdccbe3d4c08487d9b9b854bd16a4114b0fc0b1ba4601828ba6155b4fd0"} Oct 30 23:06:31.173402 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:31.173373 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:31.176157 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:31.176126 4952 patch_prober.go:29] interesting pod/router-default-5dd6f58df-tctqk container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Oct 30 23:06:31.176157 ip-10-0-130-177 kubenswrapper[4952]: [+]has-synced ok Oct 30 23:06:31.176157 ip-10-0-130-177 kubenswrapper[4952]: [+]process-running ok Oct 30 23:06:31.176157 ip-10-0-130-177 kubenswrapper[4952]: healthz check failed Oct 30 23:06:31.176343 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:31.176182 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5dd6f58df-tctqk" podUID="5f6c1ad7-10c4-4995-ae18-315d38184360" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 30 23:06:31.657317 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:31.657282 4952 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-177.us-west-1.compute.internal" event="NodeReady" Oct 30 23:06:31.830367 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:31.830326 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:32.100323 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:32.100297 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:32.174910 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:32.174887 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:32.308246 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:32.308217 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:32.517130 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:32.517054 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:32.739699 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:32.739668 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:32.948330 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:32.948299 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:33.074461 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:33.074385 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-htqtp" event={"ID":"dec2c5fa-4b9a-47ed-b005-04874761f419","Type":"ContainerStarted","Data":"9bb071fa18eb4d6aca7b943fdec8cf73642e4f7bc1df81d3c959040b324dcdbe"} Oct 30 23:06:33.074461 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:33.074438 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-htqtp" event={"ID":"dec2c5fa-4b9a-47ed-b005-04874761f419","Type":"ContainerStarted","Data":"8061869fa8bb67b432df2f0b55c13e509e935437d10c8a88fe4be224dfbefdef"} Oct 30 23:06:33.950002 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:33.949975 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9sp4c" Oct 30 23:06:34.013459 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:34.013431 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:34.077464 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:34.077439 4952 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 23:06:34.241473 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:34.241403 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:34.462118 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:34.462085 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:34.685055 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:34.685006 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:34.902319 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:34.902292 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:34.963180 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:34.963109 4952 scope.go:122] "RemoveContainer" containerID="ad8ae0b2f6a13762ab5345adf650e2cc6ec4532c73258014ffefcf23e32c4420" Oct 30 23:06:35.082502 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:35.082477 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/2.log" Oct 30 23:06:35.082655 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:35.082559 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" event={"ID":"e9a89b14-a8e8-45f2-84bd-dba21617499c","Type":"ContainerStarted","Data":"b9732a6b85ea921434b8daed4a8f98078340a29d1cff2830202767fd32d5cd59"} Oct 30 23:06:35.082844 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:35.082817 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" Oct 30 23:06:35.083728 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:35.083707 4952 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-kztwm container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.11:8443/readyz\": dial tcp 10.132.0.11:8443: connect: connection refused" start-of-body= Oct 30 23:06:35.083811 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:35.083750 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" podUID="e9a89b14-a8e8-45f2-84bd-dba21617499c" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.11:8443/readyz\": dial tcp 10.132.0.11:8443: connect: connection refused" Oct 30 23:06:35.327539 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:35.327473 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:36.090182 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:36.090155 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-kztwm" Oct 30 23:06:36.449885 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:36.449813 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:36.656710 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:36.656673 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:36.885162 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:36.885136 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:37.094706 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:37.094675 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:37.341453 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:37.341428 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:37.740925 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:37.740852 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f84d4ffd8-wx9nr"] Oct 30 23:06:37.818101 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:37.818070 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:37.975029 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:37.975000 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h5vcm\" (UID: \"f644db3c-e78a-42a8-9af3-c61194a21839\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5vcm" Oct 30 23:06:37.975232 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:37.975051 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dh6nh\" (UID: \"813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dh6nh" Oct 30 23:06:37.975232 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:37.975183 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/601f3571-aa13-43c6-8b5a-393029720fc3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:37.975358 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:37.975233 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58d18552-3df9-47f4-9291-b0e82cd954d2-metrics-tls\") pod \"dns-default-j9spz\" (UID: \"58d18552-3df9-47f4-9291-b0e82cd954d2\") " pod="openshift-dns/dns-default-j9spz" Oct 30 23:06:37.975358 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:37.975270 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0cb2a164-a8b5-4b93-b656-304550f04179-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-r26cd\" (UID: \"0cb2a164-a8b5-4b93-b656-304550f04179\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r26cd" Oct 30 23:06:37.977750 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:37.977723 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-dh6nh\" (UID: \"813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dh6nh" Oct 30 23:06:37.977874 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:37.977761 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/601f3571-aa13-43c6-8b5a-393029720fc3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lfbzq\" (UID: \"601f3571-aa13-43c6-8b5a-393029720fc3\") " pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:37.978025 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:37.978006 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58d18552-3df9-47f4-9291-b0e82cd954d2-metrics-tls\") pod \"dns-default-j9spz\" (UID: \"58d18552-3df9-47f4-9291-b0e82cd954d2\") " pod="openshift-dns/dns-default-j9spz" Oct 30 23:06:37.978117 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:37.978073 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0cb2a164-a8b5-4b93-b656-304550f04179-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-r26cd\" (UID: \"0cb2a164-a8b5-4b93-b656-304550f04179\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r26cd" Oct 30 23:06:37.978177 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:37.978132 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f644db3c-e78a-42a8-9af3-c61194a21839-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h5vcm\" (UID: \"f644db3c-e78a-42a8-9af3-c61194a21839\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5vcm" Oct 30 23:06:37.983997 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:37.983976 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lfbzq" Oct 30 23:06:37.984113 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:37.984062 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j9spz" Oct 30 23:06:38.051123 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:38.051063 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:38.271511 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:38.271442 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dh6nh" Oct 30 23:06:38.272536 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:38.272515 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5vcm" Oct 30 23:06:38.274154 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:38.274129 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-r26cd" Oct 30 23:06:38.358224 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:38.358197 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lfbzq"] Oct 30 23:06:38.361226 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:38.361202 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod601f3571_aa13_43c6_8b5a_393029720fc3.slice/crio-345589ee6914952ed8786fa94989c85a8f74dfd76a6d920f95333518b8419d29 WatchSource:0}: Error finding container 345589ee6914952ed8786fa94989c85a8f74dfd76a6d920f95333518b8419d29: Status 404 returned error can't find the container with id 345589ee6914952ed8786fa94989c85a8f74dfd76a6d920f95333518b8419d29 Oct 30 23:06:38.383122 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:38.383094 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:38.409346 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:38.409323 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-j9spz"] Oct 30 23:06:38.578241 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:38.578177 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:38.578390 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:38.578330 4952 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 23:06:38.579411 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:38.579389 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5dd6f58df-tctqk" Oct 30 23:06:38.623257 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:38.623229 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:38.644321 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:38.644293 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-r26cd"] Oct 30 23:06:38.648188 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:38.648159 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cb2a164_a8b5_4b93_b656_304550f04179.slice/crio-ca46370f0499f24fda11ef9d996842da930960ee6c69b4e28012b26d3111b9d7 WatchSource:0}: Error finding container ca46370f0499f24fda11ef9d996842da930960ee6c69b4e28012b26d3111b9d7: Status 404 returned error can't find the container with id ca46370f0499f24fda11ef9d996842da930960ee6c69b4e28012b26d3111b9d7 Oct 30 23:06:38.688145 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:38.688127 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-dh6nh"] Oct 30 23:06:38.689880 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:38.689857 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod813fd5cb_7fda_45f4_b9d0_6d8040e7f7e5.slice/crio-cc85a676fb5a143e55e44ca3717fba5f2e3e0fb88d01edb6d6f8ae5bba1bf86a WatchSource:0}: Error finding container cc85a676fb5a143e55e44ca3717fba5f2e3e0fb88d01edb6d6f8ae5bba1bf86a: Status 404 returned error can't find the container with id cc85a676fb5a143e55e44ca3717fba5f2e3e0fb88d01edb6d6f8ae5bba1bf86a Oct 30 23:06:38.704123 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:38.704102 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5vcm"] Oct 30 23:06:39.095719 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:39.095684 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5vcm" event={"ID":"f644db3c-e78a-42a8-9af3-c61194a21839","Type":"ContainerStarted","Data":"e5a4cf69de6b0e6311ca0d5b84b4b7afcc2eb842e7a66e9499247033f85464af"} Oct 30 23:06:39.097232 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:39.097196 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lfbzq" event={"ID":"601f3571-aa13-43c6-8b5a-393029720fc3","Type":"ContainerStarted","Data":"d3b091272ea866a13fab939b8fbbae5f009a9d9e44a8b4db6ef8de6fbc24a3c7"} Oct 30 23:06:39.097357 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:39.097253 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lfbzq" event={"ID":"601f3571-aa13-43c6-8b5a-393029720fc3","Type":"ContainerStarted","Data":"345589ee6914952ed8786fa94989c85a8f74dfd76a6d920f95333518b8419d29"} Oct 30 23:06:39.098356 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:39.098337 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-r26cd" event={"ID":"0cb2a164-a8b5-4b93-b656-304550f04179","Type":"ContainerStarted","Data":"ca46370f0499f24fda11ef9d996842da930960ee6c69b4e28012b26d3111b9d7"} Oct 30 23:06:39.099403 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:39.099384 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j9spz" event={"ID":"58d18552-3df9-47f4-9291-b0e82cd954d2","Type":"ContainerStarted","Data":"d59a0681d536a79be6611f829cec4a93f840fd0b8cf454126c19d26950fd265c"} Oct 30 23:06:39.100533 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:39.100513 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dh6nh" event={"ID":"813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5","Type":"ContainerStarted","Data":"cc85a676fb5a143e55e44ca3717fba5f2e3e0fb88d01edb6d6f8ae5bba1bf86a"} Oct 30 23:06:40.110668 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:40.110621 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lfbzq" event={"ID":"601f3571-aa13-43c6-8b5a-393029720fc3","Type":"ContainerStarted","Data":"eb530f18279e022cda24d91a5d34ca1cf855598530cecd346ff89e6b5e10ddb9"} Oct 30 23:06:41.116885 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:41.116834 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-r26cd" event={"ID":"0cb2a164-a8b5-4b93-b656-304550f04179","Type":"ContainerStarted","Data":"a798e1370c94bd6280e574a045d974bc7503e0e6d548dc08af2aeb8b8576d1bc"} Oct 30 23:06:41.847051 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:41.846989 4952 patch_prober.go:29] interesting pod/image-registry-9c458c7df-njhmv container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 23:06:41.847228 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:41.847059 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-9c458c7df-njhmv" podUID="3ec44853-19be-4bbb-9840-bb4c96fd2a29" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:06:42.122415 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:42.122373 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5vcm" event={"ID":"f644db3c-e78a-42a8-9af3-c61194a21839","Type":"ContainerStarted","Data":"a547b254e203b0f8a070b3c9aa3f4d5a508ea4c0d47ccbb8cebf5b6ff6e6db16"} Oct 30 23:06:42.123145 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:42.122469 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h5vcm" event={"ID":"f644db3c-e78a-42a8-9af3-c61194a21839","Type":"ContainerStarted","Data":"a60c0ca6421c76c61263ffdb5e08d9b899e76f5a614e9cd641670423c96a3199"} Oct 30 23:06:42.125197 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:42.125165 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lfbzq" event={"ID":"601f3571-aa13-43c6-8b5a-393029720fc3","Type":"ContainerStarted","Data":"b3fcc5bfa8414960cf668ff117858dd8ff0ab4d1b1b83847feafbc853ad5aa90"} Oct 30 23:06:42.127313 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:42.127290 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j9spz" event={"ID":"58d18552-3df9-47f4-9291-b0e82cd954d2","Type":"ContainerStarted","Data":"b02ca43559ff4bd418235bf2f2dc86a6ebcf1951ceebb34253537e88e0f339e2"} Oct 30 23:06:42.127430 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:42.127331 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j9spz" event={"ID":"58d18552-3df9-47f4-9291-b0e82cd954d2","Type":"ContainerStarted","Data":"b74b29c22435c9af5e7d4c99f65493f4212eaddc3ac0368f36a390fa73df82b8"} Oct 30 23:06:42.127430 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:42.127360 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-j9spz" Oct 30 23:06:42.129076 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:42.129052 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-dh6nh" event={"ID":"813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5","Type":"ContainerStarted","Data":"d9adf755df9bf90a00cdde5fa3843300995da34328497c9487498c695db3bcbc"} Oct 30 23:06:42.828230 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:42.828191 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:43.032727 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:43.032693 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:43.263278 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:43.263243 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:43.890846 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:43.890816 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:44.151864 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:44.151792 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:44.726219 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:44.726182 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:47.746261 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:47.746224 4952 patch_prober.go:29] interesting pod/image-registry-6f84d4ffd8-wx9nr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 23:06:47.746635 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:47.746277 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" podUID="8cbb1813-35e1-49cb-8718-d0ba47d2350d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:06:47.773890 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:47.773841 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:47.973627 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:47.973596 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:48.199408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:48.199376 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:48.410406 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:48.410358 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:48.623126 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:48.623077 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:49.060594 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:49.060516 4952 patch_prober.go:29] interesting pod/image-registry-9c458c7df-njhmv container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 23:06:49.060594 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:49.060569 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-9c458c7df-njhmv" podUID="3ec44853-19be-4bbb-9840-bb4c96fd2a29" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:06:51.140459 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:06:51.140423 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:06:51.846059 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:51.846015 4952 patch_prober.go:29] interesting pod/image-registry-9c458c7df-njhmv container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 23:06:51.846236 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:51.846102 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-9c458c7df-njhmv" podUID="3ec44853-19be-4bbb-9840-bb4c96fd2a29" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:06:52.135901 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:52.135817 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-j9spz" Oct 30 23:06:53.837165 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:53.837114 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:54.063912 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:54.063872 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:54.280849 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:54.280750 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:54.510098 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:54.510058 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:54.770240 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:54.770165 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:54.995305 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:54.995252 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:55.021611 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.021540 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xdlsv" Oct 30 23:06:55.126318 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.126289 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-l2rht"] Oct 30 23:06:55.129569 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.129551 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l2rht" Oct 30 23:06:55.181447 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.181421 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 23:06:55.182176 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.182153 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 23:06:55.182298 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.182236 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-s47dt\"" Oct 30 23:06:55.199334 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.199308 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17ed56a5-3a2a-494b-aaa0-a9032aa5f265-metrics-client-ca\") pod \"node-exporter-l2rht\" (UID: \"17ed56a5-3a2a-494b-aaa0-a9032aa5f265\") " pod="openshift-monitoring/node-exporter-l2rht" Oct 30 23:06:55.199446 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.199343 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/17ed56a5-3a2a-494b-aaa0-a9032aa5f265-node-exporter-accelerators-collector-config\") pod \"node-exporter-l2rht\" (UID: \"17ed56a5-3a2a-494b-aaa0-a9032aa5f265\") " pod="openshift-monitoring/node-exporter-l2rht" Oct 30 23:06:55.199446 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.199371 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17ed56a5-3a2a-494b-aaa0-a9032aa5f265-sys\") pod \"node-exporter-l2rht\" (UID: \"17ed56a5-3a2a-494b-aaa0-a9032aa5f265\") " pod="openshift-monitoring/node-exporter-l2rht" Oct 30 23:06:55.199526 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.199470 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/17ed56a5-3a2a-494b-aaa0-a9032aa5f265-node-exporter-textfile\") pod \"node-exporter-l2rht\" (UID: \"17ed56a5-3a2a-494b-aaa0-a9032aa5f265\") " pod="openshift-monitoring/node-exporter-l2rht" Oct 30 23:06:55.199526 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.199504 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/17ed56a5-3a2a-494b-aaa0-a9032aa5f265-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l2rht\" (UID: \"17ed56a5-3a2a-494b-aaa0-a9032aa5f265\") " pod="openshift-monitoring/node-exporter-l2rht" Oct 30 23:06:55.199601 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.199564 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/17ed56a5-3a2a-494b-aaa0-a9032aa5f265-root\") pod \"node-exporter-l2rht\" (UID: \"17ed56a5-3a2a-494b-aaa0-a9032aa5f265\") " pod="openshift-monitoring/node-exporter-l2rht" Oct 30 23:06:55.199601 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.199586 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/17ed56a5-3a2a-494b-aaa0-a9032aa5f265-node-exporter-tls\") pod \"node-exporter-l2rht\" (UID: \"17ed56a5-3a2a-494b-aaa0-a9032aa5f265\") " pod="openshift-monitoring/node-exporter-l2rht" Oct 30 23:06:55.199687 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.199674 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/17ed56a5-3a2a-494b-aaa0-a9032aa5f265-node-exporter-wtmp\") pod \"node-exporter-l2rht\" (UID: \"17ed56a5-3a2a-494b-aaa0-a9032aa5f265\") " pod="openshift-monitoring/node-exporter-l2rht" Oct 30 23:06:55.199736 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.199700 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9p7q\" (UniqueName: \"kubernetes.io/projected/17ed56a5-3a2a-494b-aaa0-a9032aa5f265-kube-api-access-d9p7q\") pod \"node-exporter-l2rht\" (UID: \"17ed56a5-3a2a-494b-aaa0-a9032aa5f265\") " pod="openshift-monitoring/node-exporter-l2rht" Oct 30 23:06:55.221447 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.221423 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:55.262316 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.262291 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 23:06:55.262549 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.262533 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 23:06:55.300911 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.300851 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/17ed56a5-3a2a-494b-aaa0-a9032aa5f265-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l2rht\" (UID: \"17ed56a5-3a2a-494b-aaa0-a9032aa5f265\") " pod="openshift-monitoring/node-exporter-l2rht" Oct 30 23:06:55.300911 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.300888 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/17ed56a5-3a2a-494b-aaa0-a9032aa5f265-root\") pod \"node-exporter-l2rht\" (UID: \"17ed56a5-3a2a-494b-aaa0-a9032aa5f265\") " pod="openshift-monitoring/node-exporter-l2rht" Oct 30 23:06:55.301095 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.300919 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/17ed56a5-3a2a-494b-aaa0-a9032aa5f265-node-exporter-tls\") pod \"node-exporter-l2rht\" (UID: \"17ed56a5-3a2a-494b-aaa0-a9032aa5f265\") " pod="openshift-monitoring/node-exporter-l2rht" Oct 30 23:06:55.301095 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.300966 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/17ed56a5-3a2a-494b-aaa0-a9032aa5f265-node-exporter-wtmp\") pod \"node-exporter-l2rht\" (UID: \"17ed56a5-3a2a-494b-aaa0-a9032aa5f265\") " pod="openshift-monitoring/node-exporter-l2rht" Oct 30 23:06:55.301095 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.300973 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/17ed56a5-3a2a-494b-aaa0-a9032aa5f265-root\") pod \"node-exporter-l2rht\" (UID: \"17ed56a5-3a2a-494b-aaa0-a9032aa5f265\") " pod="openshift-monitoring/node-exporter-l2rht" Oct 30 23:06:55.301095 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.300995 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9p7q\" (UniqueName: \"kubernetes.io/projected/17ed56a5-3a2a-494b-aaa0-a9032aa5f265-kube-api-access-d9p7q\") pod \"node-exporter-l2rht\" (UID: \"17ed56a5-3a2a-494b-aaa0-a9032aa5f265\") " pod="openshift-monitoring/node-exporter-l2rht" Oct 30 23:06:55.301095 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.301057 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17ed56a5-3a2a-494b-aaa0-a9032aa5f265-metrics-client-ca\") pod \"node-exporter-l2rht\" (UID: \"17ed56a5-3a2a-494b-aaa0-a9032aa5f265\") " pod="openshift-monitoring/node-exporter-l2rht" Oct 30 23:06:55.301364 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.301095 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/17ed56a5-3a2a-494b-aaa0-a9032aa5f265-node-exporter-accelerators-collector-config\") pod \"node-exporter-l2rht\" (UID: \"17ed56a5-3a2a-494b-aaa0-a9032aa5f265\") " pod="openshift-monitoring/node-exporter-l2rht" Oct 30 23:06:55.301364 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.301132 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/17ed56a5-3a2a-494b-aaa0-a9032aa5f265-node-exporter-wtmp\") pod \"node-exporter-l2rht\" (UID: \"17ed56a5-3a2a-494b-aaa0-a9032aa5f265\") " pod="openshift-monitoring/node-exporter-l2rht" Oct 30 23:06:55.301364 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.301138 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17ed56a5-3a2a-494b-aaa0-a9032aa5f265-sys\") pod \"node-exporter-l2rht\" (UID: \"17ed56a5-3a2a-494b-aaa0-a9032aa5f265\") " pod="openshift-monitoring/node-exporter-l2rht" Oct 30 23:06:55.301364 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.301191 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17ed56a5-3a2a-494b-aaa0-a9032aa5f265-sys\") pod \"node-exporter-l2rht\" (UID: \"17ed56a5-3a2a-494b-aaa0-a9032aa5f265\") " pod="openshift-monitoring/node-exporter-l2rht" Oct 30 23:06:55.301364 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.301211 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/17ed56a5-3a2a-494b-aaa0-a9032aa5f265-node-exporter-textfile\") pod \"node-exporter-l2rht\" (UID: \"17ed56a5-3a2a-494b-aaa0-a9032aa5f265\") " pod="openshift-monitoring/node-exporter-l2rht" Oct 30 23:06:55.301584 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.301493 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/17ed56a5-3a2a-494b-aaa0-a9032aa5f265-node-exporter-textfile\") pod \"node-exporter-l2rht\" (UID: \"17ed56a5-3a2a-494b-aaa0-a9032aa5f265\") " pod="openshift-monitoring/node-exporter-l2rht" Oct 30 23:06:55.301704 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.301682 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17ed56a5-3a2a-494b-aaa0-a9032aa5f265-metrics-client-ca\") pod \"node-exporter-l2rht\" (UID: \"17ed56a5-3a2a-494b-aaa0-a9032aa5f265\") " pod="openshift-monitoring/node-exporter-l2rht" Oct 30 23:06:55.301749 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.301736 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/17ed56a5-3a2a-494b-aaa0-a9032aa5f265-node-exporter-accelerators-collector-config\") pod \"node-exporter-l2rht\" (UID: \"17ed56a5-3a2a-494b-aaa0-a9032aa5f265\") " pod="openshift-monitoring/node-exporter-l2rht" Oct 30 23:06:55.303258 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.303238 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/17ed56a5-3a2a-494b-aaa0-a9032aa5f265-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l2rht\" (UID: \"17ed56a5-3a2a-494b-aaa0-a9032aa5f265\") " pod="openshift-monitoring/node-exporter-l2rht" Oct 30 23:06:55.303475 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.303458 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/17ed56a5-3a2a-494b-aaa0-a9032aa5f265-node-exporter-tls\") pod \"node-exporter-l2rht\" (UID: \"17ed56a5-3a2a-494b-aaa0-a9032aa5f265\") " pod="openshift-monitoring/node-exporter-l2rht" Oct 30 23:06:55.357478 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.357455 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9p7q\" (UniqueName: \"kubernetes.io/projected/17ed56a5-3a2a-494b-aaa0-a9032aa5f265-kube-api-access-d9p7q\") pod \"node-exporter-l2rht\" (UID: \"17ed56a5-3a2a-494b-aaa0-a9032aa5f265\") " pod="openshift-monitoring/node-exporter-l2rht" Oct 30 23:06:55.432372 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.432339 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:55.443010 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.442989 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l2rht" Oct 30 23:06:55.453197 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:06:55.453170 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ed56a5_3a2a_494b_aaa0_a9032aa5f265.slice/crio-fcbdf601bc6a11ba3b8e7d95c51556b99cb3559b5ea49c006d58c39cada8e52f WatchSource:0}: Error finding container fcbdf601bc6a11ba3b8e7d95c51556b99cb3559b5ea49c006d58c39cada8e52f: Status 404 returned error can't find the container with id fcbdf601bc6a11ba3b8e7d95c51556b99cb3559b5ea49c006d58c39cada8e52f Oct 30 23:06:55.658720 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:55.658686 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:56.081243 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:56.081167 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:56.178607 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:56.178576 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l2rht" event={"ID":"17ed56a5-3a2a-494b-aaa0-a9032aa5f265","Type":"ContainerStarted","Data":"fcbdf601bc6a11ba3b8e7d95c51556b99cb3559b5ea49c006d58c39cada8e52f"} Oct 30 23:06:56.482320 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:56.482287 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:56.804453 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:56.804373 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:57.183956 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:57.183919 4952 generic.go:356] "Generic (PLEG): container finished" podID="17ed56a5-3a2a-494b-aaa0-a9032aa5f265" containerID="d1b6cd286dd67d1fb108ee4f406a701d1ea42eb4f799feeb8dd4afa0d7340358" exitCode=0 Oct 30 23:06:57.184396 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:57.183992 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l2rht" event={"ID":"17ed56a5-3a2a-494b-aaa0-a9032aa5f265","Type":"ContainerDied","Data":"d1b6cd286dd67d1fb108ee4f406a701d1ea42eb4f799feeb8dd4afa0d7340358"} Oct 30 23:06:57.345252 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:57.345218 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:57.745989 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:57.745965 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:06:58.189737 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:58.189698 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l2rht" event={"ID":"17ed56a5-3a2a-494b-aaa0-a9032aa5f265","Type":"ContainerStarted","Data":"8abcf515d62a7382d1525b772be598ca1dbfeafcea87e707af541d79088ea92e"} Oct 30 23:06:58.190095 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:58.189760 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l2rht" event={"ID":"17ed56a5-3a2a-494b-aaa0-a9032aa5f265","Type":"ContainerStarted","Data":"e17009184569da90f514802f94dadab1dd483d593c9a2d5a4d7532f000580196"} Oct 30 23:06:58.308124 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:58.308068 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-l2rht" podStartSLOduration=1.802102772 podStartE2EDuration="3.308027836s" totalImagesPullingTime="1.431149459s" totalInitContainerRuntime="74.775605ms" isStatefulPod=true podCreationTimestamp="2026-10-30 23:06:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:06:58.306327648 +0000 UTC m=+37.450157854" watchObservedRunningTime="2026-10-30 23:06:58.308027836 +0000 UTC m=+37.451858041" Oct 30 23:06:59.061167 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:59.061134 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:06:59.255135 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:59.255097 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:59.506307 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:59.506268 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:59.733442 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:59.733411 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:06:59.967254 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:06:59.967168 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:07:01.282779 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:01.282745 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:07:01.522249 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:01.522200 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:07:02.760940 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:02.760894 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" podUID="8cbb1813-35e1-49cb-8718-d0ba47d2350d" containerName="registry" containerID="cri-o://1fa807a31856c6c15abfcb704388b03330af1626dd7ea009cf1fb5b2bbcb5a0b" gracePeriod=30 Oct 30 23:07:02.860484 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:02.860451 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:07:03.001969 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.001948 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:07:03.083873 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.083804 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:07:03.167207 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.167183 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-kube-api-access-tpjxg\" (UniqueName: \"kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-kube-api-access-tpjxg\") pod \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " Oct 30 23:07:03.167333 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.167223 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-tls\" (UniqueName: \"kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-tls\") pod \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " Oct 30 23:07:03.167333 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.167251 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cbb1813-35e1-49cb-8718-d0ba47d2350d-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8cbb1813-35e1-49cb-8718-d0ba47d2350d-installation-pull-secrets\") pod \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " Oct 30 23:07:03.167333 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.167291 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cbb1813-35e1-49cb-8718-d0ba47d2350d-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8cbb1813-35e1-49cb-8718-d0ba47d2350d-image-registry-private-configuration\") pod \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " Oct 30 23:07:03.167333 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.167321 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cbb1813-35e1-49cb-8718-d0ba47d2350d-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8cbb1813-35e1-49cb-8718-d0ba47d2350d-trusted-ca\") pod \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " Oct 30 23:07:03.167555 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.167356 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cbb1813-35e1-49cb-8718-d0ba47d2350d-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8cbb1813-35e1-49cb-8718-d0ba47d2350d-ca-trust-extracted\") pod \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " Oct 30 23:07:03.167555 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.167401 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-certificates\") pod \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " Oct 30 23:07:03.167555 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.167441 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-bound-sa-token\") pod \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\" (UID: \"8cbb1813-35e1-49cb-8718-d0ba47d2350d\") " Oct 30 23:07:03.168186 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.168099 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cbb1813-35e1-49cb-8718-d0ba47d2350d-trusted-ca" pod "8cbb1813-35e1-49cb-8718-d0ba47d2350d" (UID: "8cbb1813-35e1-49cb-8718-d0ba47d2350d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:07:03.168601 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.168568 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-certificates" pod "8cbb1813-35e1-49cb-8718-d0ba47d2350d" (UID: "8cbb1813-35e1-49cb-8718-d0ba47d2350d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:07:03.170159 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.170119 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cbb1813-35e1-49cb-8718-d0ba47d2350d-image-registry-private-configuration" pod "8cbb1813-35e1-49cb-8718-d0ba47d2350d" (UID: "8cbb1813-35e1-49cb-8718-d0ba47d2350d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:07:03.170270 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.170169 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-kube-api-access-tpjxg" pod "8cbb1813-35e1-49cb-8718-d0ba47d2350d" (UID: "8cbb1813-35e1-49cb-8718-d0ba47d2350d"). InnerVolumeSpecName "kube-api-access-tpjxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:07:03.170270 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.170215 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cbb1813-35e1-49cb-8718-d0ba47d2350d-installation-pull-secrets" pod "8cbb1813-35e1-49cb-8718-d0ba47d2350d" (UID: "8cbb1813-35e1-49cb-8718-d0ba47d2350d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:07:03.170270 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.170259 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-tls" pod "8cbb1813-35e1-49cb-8718-d0ba47d2350d" (UID: "8cbb1813-35e1-49cb-8718-d0ba47d2350d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:07:03.170384 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.170268 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-bound-sa-token" pod "8cbb1813-35e1-49cb-8718-d0ba47d2350d" (UID: "8cbb1813-35e1-49cb-8718-d0ba47d2350d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:07:03.174958 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.174931 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cbb1813-35e1-49cb-8718-d0ba47d2350d-ca-trust-extracted" pod "8cbb1813-35e1-49cb-8718-d0ba47d2350d" (UID: "8cbb1813-35e1-49cb-8718-d0ba47d2350d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:07:03.209562 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.209535 4952 generic.go:356] "Generic (PLEG): container finished" podID="8cbb1813-35e1-49cb-8718-d0ba47d2350d" containerID="1fa807a31856c6c15abfcb704388b03330af1626dd7ea009cf1fb5b2bbcb5a0b" exitCode=0 Oct 30 23:07:03.209680 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.209588 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" Oct 30 23:07:03.209680 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.209633 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" event={"ID":"8cbb1813-35e1-49cb-8718-d0ba47d2350d","Type":"ContainerDied","Data":"1fa807a31856c6c15abfcb704388b03330af1626dd7ea009cf1fb5b2bbcb5a0b"} Oct 30 23:07:03.209765 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.209688 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f84d4ffd8-wx9nr" event={"ID":"8cbb1813-35e1-49cb-8718-d0ba47d2350d","Type":"ContainerDied","Data":"c33608cc1d1779d0316f484afb4803b78407e567af1f6ed38140bed3d7f29b74"} Oct 30 23:07:03.209765 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.209709 4952 scope.go:122] "RemoveContainer" containerID="1fa807a31856c6c15abfcb704388b03330af1626dd7ea009cf1fb5b2bbcb5a0b" Oct 30 23:07:03.218154 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.218023 4952 scope.go:122] "RemoveContainer" containerID="1fa807a31856c6c15abfcb704388b03330af1626dd7ea009cf1fb5b2bbcb5a0b" Oct 30 23:07:03.218320 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:07:03.218301 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fa807a31856c6c15abfcb704388b03330af1626dd7ea009cf1fb5b2bbcb5a0b\": container with ID starting with 1fa807a31856c6c15abfcb704388b03330af1626dd7ea009cf1fb5b2bbcb5a0b not found: ID does not exist" containerID="1fa807a31856c6c15abfcb704388b03330af1626dd7ea009cf1fb5b2bbcb5a0b" Oct 30 23:07:03.218363 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.218329 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fa807a31856c6c15abfcb704388b03330af1626dd7ea009cf1fb5b2bbcb5a0b"} err="failed to get container status \"1fa807a31856c6c15abfcb704388b03330af1626dd7ea009cf1fb5b2bbcb5a0b\": rpc error: code = NotFound desc = could not find container \"1fa807a31856c6c15abfcb704388b03330af1626dd7ea009cf1fb5b2bbcb5a0b\": container with ID starting with 1fa807a31856c6c15abfcb704388b03330af1626dd7ea009cf1fb5b2bbcb5a0b not found: ID does not exist" Oct 30 23:07:03.267988 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.267962 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpjxg\" (UniqueName: \"kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-kube-api-access-tpjxg\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:07:03.267988 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.267984 4952 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-tls\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:07:03.268142 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.267993 4952 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8cbb1813-35e1-49cb-8718-d0ba47d2350d-installation-pull-secrets\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:07:03.268142 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.268002 4952 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8cbb1813-35e1-49cb-8718-d0ba47d2350d-image-registry-private-configuration\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:07:03.268142 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.268011 4952 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8cbb1813-35e1-49cb-8718-d0ba47d2350d-trusted-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:07:03.268142 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.268020 4952 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8cbb1813-35e1-49cb-8718-d0ba47d2350d-ca-trust-extracted\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:07:03.268142 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.268029 4952 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8cbb1813-35e1-49cb-8718-d0ba47d2350d-registry-certificates\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:07:03.268142 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.268058 4952 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cbb1813-35e1-49cb-8718-d0ba47d2350d-bound-sa-token\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:07:03.375960 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.375933 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f84d4ffd8-wx9nr"] Oct 30 23:07:03.382959 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.382940 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6f84d4ffd8-wx9nr"] Oct 30 23:07:03.684475 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.684375 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:07:03.919655 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:03.919605 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:07:04.322517 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:04.322487 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:07:04.967483 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:04.967451 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cbb1813-35e1-49cb-8718-d0ba47d2350d" path="/var/lib/kubelet/pods/8cbb1813-35e1-49cb-8718-d0ba47d2350d/volumes" Oct 30 23:07:05.927628 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:05.927594 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:07:06.133528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:06.133496 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:07:08.151353 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:08.151320 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-9c458c7df-njhmv"] Oct 30 23:07:08.352150 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:08.352113 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:07:08.591923 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:08.591889 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:07:08.856577 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:08.856499 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:07:10.464083 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:10.463993 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:07:10.721459 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:10.721347 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:07:12.215253 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:12.215224 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:07:12.433614 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:12.433579 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:07:17.789511 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:17.789470 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:07:18.037884 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:18.037853 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:07:21.140666 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:07:21.140630 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:07:23.934574 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:23.934537 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:07:24.200317 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:24.200225 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:07:24.416488 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:24.416455 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:07:26.366563 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:26.366528 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:07:26.596357 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:26.596323 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:07:26.821348 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:26.821274 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:07:27.054344 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:27.054313 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:07:32.122204 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:32.122171 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:07:32.317349 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:32.317318 4952 generic.go:356] "Generic (PLEG): container finished" podID="edc2af65-1515-4742-89e1-b49814016b09" containerID="14104234ddbdd88cfcb38a94ecf6cfe9de0320fca2ed0f5b273eda0f508d680c" exitCode=0 Oct 30 23:07:32.317518 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:32.317395 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j7f9n" event={"ID":"edc2af65-1515-4742-89e1-b49814016b09","Type":"ContainerDied","Data":"14104234ddbdd88cfcb38a94ecf6cfe9de0320fca2ed0f5b273eda0f508d680c"} Oct 30 23:07:32.317710 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:32.317695 4952 scope.go:122] "RemoveContainer" containerID="14104234ddbdd88cfcb38a94ecf6cfe9de0320fca2ed0f5b273eda0f508d680c" Oct 30 23:07:32.405837 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:32.405805 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:07:32.780444 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:32.780369 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:07:33.019922 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:33.019890 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:07:33.174663 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:33.174617 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-9c458c7df-njhmv" podUID="3ec44853-19be-4bbb-9840-bb4c96fd2a29" containerName="registry" containerID="cri-o://fd91ccb839d39661c4e9fd1f00952839572ee83292febfc3d89fce1f77ab9524" gracePeriod=30 Oct 30 23:07:33.322859 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:33.322830 4952 generic.go:356] "Generic (PLEG): container finished" podID="3ec44853-19be-4bbb-9840-bb4c96fd2a29" containerID="fd91ccb839d39661c4e9fd1f00952839572ee83292febfc3d89fce1f77ab9524" exitCode=0 Oct 30 23:07:33.323009 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:33.322903 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9c458c7df-njhmv" event={"ID":"3ec44853-19be-4bbb-9840-bb4c96fd2a29","Type":"ContainerDied","Data":"fd91ccb839d39661c4e9fd1f00952839572ee83292febfc3d89fce1f77ab9524"} Oct 30 23:07:33.324808 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:33.324782 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j7f9n" event={"ID":"edc2af65-1515-4742-89e1-b49814016b09","Type":"ContainerStarted","Data":"d31bb88abab53f17268191063c2cc65dde824be1bfd73ea1cde1a6270c9c019a"} Oct 30 23:07:33.433514 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:33.433462 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:07:33.492217 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:33.492191 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-tls\" (UniqueName: \"kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-tls\") pod \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " Oct 30 23:07:33.492391 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:33.492234 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-bound-sa-token\") pod \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " Oct 30 23:07:33.492391 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:33.492270 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ec44853-19be-4bbb-9840-bb4c96fd2a29-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ec44853-19be-4bbb-9840-bb4c96fd2a29-installation-pull-secrets\") pod \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " Oct 30 23:07:33.492391 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:33.492385 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ec44853-19be-4bbb-9840-bb4c96fd2a29-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ec44853-19be-4bbb-9840-bb4c96fd2a29-image-registry-private-configuration\") pod \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " Oct 30 23:07:33.492559 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:33.492426 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-kube-api-access-pvnfz\" (UniqueName: \"kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-kube-api-access-pvnfz\") pod \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " Oct 30 23:07:33.492559 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:33.492469 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-certificates\") pod \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " Oct 30 23:07:33.492559 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:33.492512 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ec44853-19be-4bbb-9840-bb4c96fd2a29-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ec44853-19be-4bbb-9840-bb4c96fd2a29-trusted-ca\") pod \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " Oct 30 23:07:33.492559 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:33.492544 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ec44853-19be-4bbb-9840-bb4c96fd2a29-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ec44853-19be-4bbb-9840-bb4c96fd2a29-ca-trust-extracted\") pod \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\" (UID: \"3ec44853-19be-4bbb-9840-bb4c96fd2a29\") " Oct 30 23:07:33.493425 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:33.493355 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec44853-19be-4bbb-9840-bb4c96fd2a29-trusted-ca" pod "3ec44853-19be-4bbb-9840-bb4c96fd2a29" (UID: "3ec44853-19be-4bbb-9840-bb4c96fd2a29"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:07:33.493747 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:33.493711 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-certificates" pod "3ec44853-19be-4bbb-9840-bb4c96fd2a29" (UID: "3ec44853-19be-4bbb-9840-bb4c96fd2a29"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:07:33.495247 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:33.495027 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-tls" pod "3ec44853-19be-4bbb-9840-bb4c96fd2a29" (UID: "3ec44853-19be-4bbb-9840-bb4c96fd2a29"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:07:33.495548 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:33.495436 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-bound-sa-token" pod "3ec44853-19be-4bbb-9840-bb4c96fd2a29" (UID: "3ec44853-19be-4bbb-9840-bb4c96fd2a29"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:07:33.495548 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:33.495507 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ec44853-19be-4bbb-9840-bb4c96fd2a29-installation-pull-secrets" pod "3ec44853-19be-4bbb-9840-bb4c96fd2a29" (UID: "3ec44853-19be-4bbb-9840-bb4c96fd2a29"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:07:33.495548 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:33.495541 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-kube-api-access-pvnfz" pod "3ec44853-19be-4bbb-9840-bb4c96fd2a29" (UID: "3ec44853-19be-4bbb-9840-bb4c96fd2a29"). InnerVolumeSpecName "kube-api-access-pvnfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:07:33.495756 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:33.495712 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ec44853-19be-4bbb-9840-bb4c96fd2a29-image-registry-private-configuration" pod "3ec44853-19be-4bbb-9840-bb4c96fd2a29" (UID: "3ec44853-19be-4bbb-9840-bb4c96fd2a29"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:07:33.500711 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:33.500691 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ec44853-19be-4bbb-9840-bb4c96fd2a29-ca-trust-extracted" pod "3ec44853-19be-4bbb-9840-bb4c96fd2a29" (UID: "3ec44853-19be-4bbb-9840-bb4c96fd2a29"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:07:33.593657 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:33.593626 4952 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-tls\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:07:33.593657 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:33.593651 4952 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-bound-sa-token\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:07:33.593657 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:33.593661 4952 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ec44853-19be-4bbb-9840-bb4c96fd2a29-installation-pull-secrets\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:07:33.593855 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:33.593672 4952 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ec44853-19be-4bbb-9840-bb4c96fd2a29-image-registry-private-configuration\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:07:33.593855 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:33.593682 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvnfz\" (UniqueName: \"kubernetes.io/projected/3ec44853-19be-4bbb-9840-bb4c96fd2a29-kube-api-access-pvnfz\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:07:33.593855 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:33.593691 4952 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ec44853-19be-4bbb-9840-bb4c96fd2a29-registry-certificates\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:07:33.593855 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:33.593700 4952 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ec44853-19be-4bbb-9840-bb4c96fd2a29-trusted-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:07:33.593855 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:33.593708 4952 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ec44853-19be-4bbb-9840-bb4c96fd2a29-ca-trust-extracted\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:07:34.329841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:34.329804 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9c458c7df-njhmv" event={"ID":"3ec44853-19be-4bbb-9840-bb4c96fd2a29","Type":"ContainerDied","Data":"b11590e240703cee89609a7a7f0225d852c3b0ebc0dbb695aa87035b33d995e2"} Oct 30 23:07:34.330316 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:34.329847 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9c458c7df-njhmv" Oct 30 23:07:34.330316 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:34.329873 4952 scope.go:122] "RemoveContainer" containerID="fd91ccb839d39661c4e9fd1f00952839572ee83292febfc3d89fce1f77ab9524" Oct 30 23:07:34.542372 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:34.542340 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-9c458c7df-njhmv"] Oct 30 23:07:34.592360 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:34.592332 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-9c458c7df-njhmv"] Oct 30 23:07:34.967762 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:34.967682 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ec44853-19be-4bbb-9840-bb4c96fd2a29" path="/var/lib/kubelet/pods/3ec44853-19be-4bbb-9840-bb4c96fd2a29/volumes" Oct 30 23:07:42.364659 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:42.364628 4952 generic.go:356] "Generic (PLEG): container finished" podID="6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893" containerID="1da7179d70e8597de30ed974576439dd55d684f2d10d3f4e1b4be85ce0ef0a37" exitCode=0 Oct 30 23:07:42.365089 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:42.364696 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chxn6" event={"ID":"6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893","Type":"ContainerDied","Data":"1da7179d70e8597de30ed974576439dd55d684f2d10d3f4e1b4be85ce0ef0a37"} Oct 30 23:07:42.365089 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:42.364961 4952 scope.go:122] "RemoveContainer" containerID="1da7179d70e8597de30ed974576439dd55d684f2d10d3f4e1b4be85ce0ef0a37" Oct 30 23:07:42.722736 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:42.722659 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:07:42.952637 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:42.952605 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:07:43.370194 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:43.370155 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chxn6" event={"ID":"6b1f42be-1ef5-43bb-a17f-b3b1a9b0f893","Type":"ContainerStarted","Data":"3eab53fe345881590c7c47c1364c3c222927ed63efa2c3dbfdb78d83f0cd4beb"} Oct 30 23:07:51.139567 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:07:51.139529 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:07:51.404009 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:51.403935 4952 generic.go:356] "Generic (PLEG): container finished" podID="ef598d5f-a066-4b2d-96bd-fbfc47741dbb" containerID="18edf6ff4c8ba28a7b3c0865d398080e669631865a0de77558bb5c5c4457426b" exitCode=0 Oct 30 23:07:51.404170 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:51.404008 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-g97cr" event={"ID":"ef598d5f-a066-4b2d-96bd-fbfc47741dbb","Type":"ContainerDied","Data":"18edf6ff4c8ba28a7b3c0865d398080e669631865a0de77558bb5c5c4457426b"} Oct 30 23:07:51.404324 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:51.404310 4952 scope.go:122] "RemoveContainer" containerID="18edf6ff4c8ba28a7b3c0865d398080e669631865a0de77558bb5c5c4457426b" Oct 30 23:07:52.409768 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:07:52.409738 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-g97cr" event={"ID":"ef598d5f-a066-4b2d-96bd-fbfc47741dbb","Type":"ContainerStarted","Data":"9c660d79ff7c88da7f6741d841427b16cb743b2b4deb6b420c08a921a17ef4fb"} Oct 30 23:08:10.982109 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:08:10.982079 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:08:21.140486 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:08:21.140452 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:08:35.368557 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:08:35.368481 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:08:51.141384 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:08:51.141341 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:09:11.326026 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:09:11.325991 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:09:21.140494 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:09:21.140457 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:09:46.075402 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:09:46.075361 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:09:46.294931 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:09:46.294894 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:09:50.022119 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:09:50.022086 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:09:50.246433 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:09:50.246397 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:09:51.140770 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:09:51.140737 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:10:02.761564 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:10:02.761477 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:10:02.974202 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:10:02.974168 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:10:11.660369 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:10:11.660336 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:10:21.140656 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:10:21.140616 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:10:27.900894 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:10:27.900854 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:10:51.142473 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:10:51.142437 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:11:12.034148 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:12.034117 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:11:20.983341 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:20.983309 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/2.log" Oct 30 23:11:20.983862 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:20.983546 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/2.log" Oct 30 23:11:20.989229 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:20.989207 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 30 23:11:20.989618 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:20.989594 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 30 23:11:21.143798 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:11:21.143766 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:11:25.577202 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:25.577168 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:11:25.846456 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:25.846424 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:11:27.227047 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:27.227001 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:11:27.469731 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:27.469684 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:11:27.831535 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:27.831502 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac2glm"] Oct 30 23:11:27.831867 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:27.831848 4952 cpu_manager.go:414] "containerMap: removing container" podUID="3ec44853-19be-4bbb-9840-bb4c96fd2a29" containerName="registry" Oct 30 23:11:27.831950 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:27.831868 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ec44853-19be-4bbb-9840-bb4c96fd2a29" containerName="registry" Oct 30 23:11:27.831950 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:27.831886 4952 cpu_manager.go:414] "containerMap: removing container" podUID="8cbb1813-35e1-49cb-8718-d0ba47d2350d" containerName="registry" Oct 30 23:11:27.831950 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:27.831892 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cbb1813-35e1-49cb-8718-d0ba47d2350d" containerName="registry" Oct 30 23:11:27.832131 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:27.831994 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ec44853-19be-4bbb-9840-bb4c96fd2a29" containerName="registry" Oct 30 23:11:27.832131 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:27.832008 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cbb1813-35e1-49cb-8718-d0ba47d2350d" containerName="registry" Oct 30 23:11:27.835232 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:27.835210 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac2glm" Oct 30 23:11:27.902217 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:27.902186 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac2glm\" (UID: \"8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac2glm" Oct 30 23:11:27.902352 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:27.902237 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac2glm\" (UID: \"8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac2glm" Oct 30 23:11:27.902352 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:27.902334 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xtlg\" (UniqueName: \"kubernetes.io/projected/8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3-kube-api-access-2xtlg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac2glm\" (UID: \"8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac2glm" Oct 30 23:11:27.936890 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:27.936862 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 23:11:27.936890 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:27.936862 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 23:11:27.937973 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:27.937957 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac2glm"] Oct 30 23:11:27.938714 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:27.938695 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78trw2"] Oct 30 23:11:27.939861 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:27.939842 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-bswfh\"" Oct 30 23:11:27.942024 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:27.942009 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78trw2" Oct 30 23:11:28.003235 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.003215 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78trw2\" (UID: \"b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78trw2" Oct 30 23:11:28.003342 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.003257 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xtlg\" (UniqueName: \"kubernetes.io/projected/8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3-kube-api-access-2xtlg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac2glm\" (UID: \"8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac2glm" Oct 30 23:11:28.003342 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.003287 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac2glm\" (UID: \"8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac2glm" Oct 30 23:11:28.003462 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.003350 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78trw2\" (UID: \"b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78trw2" Oct 30 23:11:28.003462 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.003391 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac2glm\" (UID: \"8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac2glm" Oct 30 23:11:28.003462 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.003418 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8nf8\" (UniqueName: \"kubernetes.io/projected/b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41-kube-api-access-b8nf8\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78trw2\" (UID: \"b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78trw2" Oct 30 23:11:28.003709 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.003691 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac2glm\" (UID: \"8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac2glm" Oct 30 23:11:28.003780 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.003725 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac2glm\" (UID: \"8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac2glm" Oct 30 23:11:28.007202 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.007178 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78trw2"] Oct 30 23:11:28.023974 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.023950 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-5xfs9\"" Oct 30 23:11:28.060926 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.060903 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft"] Oct 30 23:11:28.064249 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.064229 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft" Oct 30 23:11:28.064338 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.064273 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xtlg\" (UniqueName: \"kubernetes.io/projected/8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3-kube-api-access-2xtlg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac2glm\" (UID: \"8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac2glm" Oct 30 23:11:28.104454 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.104423 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7656f\" (UniqueName: \"kubernetes.io/projected/91651ce2-4673-4ced-b26d-e94ff8829d11-kube-api-access-7656f\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft\" (UID: \"91651ce2-4673-4ced-b26d-e94ff8829d11\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft" Oct 30 23:11:28.104566 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.104466 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78trw2\" (UID: \"b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78trw2" Oct 30 23:11:28.104566 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.104494 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/91651ce2-4673-4ced-b26d-e94ff8829d11-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft\" (UID: \"91651ce2-4673-4ced-b26d-e94ff8829d11\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft" Oct 30 23:11:28.104671 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.104649 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/91651ce2-4673-4ced-b26d-e94ff8829d11-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft\" (UID: \"91651ce2-4673-4ced-b26d-e94ff8829d11\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft" Oct 30 23:11:28.104713 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.104690 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8nf8\" (UniqueName: \"kubernetes.io/projected/b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41-kube-api-access-b8nf8\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78trw2\" (UID: \"b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78trw2" Oct 30 23:11:28.104755 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.104717 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78trw2\" (UID: \"b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78trw2" Oct 30 23:11:28.104796 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.104770 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78trw2\" (UID: \"b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78trw2" Oct 30 23:11:28.105020 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.105005 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78trw2\" (UID: \"b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78trw2" Oct 30 23:11:28.120930 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.120906 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-79kmc\"" Oct 30 23:11:28.121874 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.121847 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft"] Oct 30 23:11:28.125171 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.125151 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6pcxm"] Oct 30 23:11:28.128473 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.128456 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6pcxm" Oct 30 23:11:28.150394 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.150375 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac2glm" Oct 30 23:11:28.163465 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.163448 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8nf8\" (UniqueName: \"kubernetes.io/projected/b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41-kube-api-access-b8nf8\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78trw2\" (UID: \"b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78trw2" Oct 30 23:11:28.185732 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.185698 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6pcxm"] Oct 30 23:11:28.205574 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.205538 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7656f\" (UniqueName: \"kubernetes.io/projected/91651ce2-4673-4ced-b26d-e94ff8829d11-kube-api-access-7656f\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft\" (UID: \"91651ce2-4673-4ced-b26d-e94ff8829d11\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft" Oct 30 23:11:28.205674 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.205601 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/91651ce2-4673-4ced-b26d-e94ff8829d11-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft\" (UID: \"91651ce2-4673-4ced-b26d-e94ff8829d11\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft" Oct 30 23:11:28.205674 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.205647 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae856da3-bc5b-43df-a4ec-45cae56db52f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6pcxm\" (UID: \"ae856da3-bc5b-43df-a4ec-45cae56db52f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6pcxm" Oct 30 23:11:28.205753 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.205676 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/91651ce2-4673-4ced-b26d-e94ff8829d11-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft\" (UID: \"91651ce2-4673-4ced-b26d-e94ff8829d11\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft" Oct 30 23:11:28.205753 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.205726 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae856da3-bc5b-43df-a4ec-45cae56db52f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6pcxm\" (UID: \"ae856da3-bc5b-43df-a4ec-45cae56db52f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6pcxm" Oct 30 23:11:28.205825 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.205789 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64mrb\" (UniqueName: \"kubernetes.io/projected/ae856da3-bc5b-43df-a4ec-45cae56db52f-kube-api-access-64mrb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6pcxm\" (UID: \"ae856da3-bc5b-43df-a4ec-45cae56db52f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6pcxm" Oct 30 23:11:28.206535 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.206513 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/91651ce2-4673-4ced-b26d-e94ff8829d11-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft\" (UID: \"91651ce2-4673-4ced-b26d-e94ff8829d11\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft" Oct 30 23:11:28.206987 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.206784 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/91651ce2-4673-4ced-b26d-e94ff8829d11-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft\" (UID: \"91651ce2-4673-4ced-b26d-e94ff8829d11\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft" Oct 30 23:11:28.220755 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.220729 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-gkdpq\"" Oct 30 23:11:28.257177 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.257156 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78trw2" Oct 30 23:11:28.263492 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.263472 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7656f\" (UniqueName: \"kubernetes.io/projected/91651ce2-4673-4ced-b26d-e94ff8829d11-kube-api-access-7656f\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft\" (UID: \"91651ce2-4673-4ced-b26d-e94ff8829d11\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft" Oct 30 23:11:28.308542 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.307170 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64mrb\" (UniqueName: \"kubernetes.io/projected/ae856da3-bc5b-43df-a4ec-45cae56db52f-kube-api-access-64mrb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6pcxm\" (UID: \"ae856da3-bc5b-43df-a4ec-45cae56db52f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6pcxm" Oct 30 23:11:28.308542 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.307262 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae856da3-bc5b-43df-a4ec-45cae56db52f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6pcxm\" (UID: \"ae856da3-bc5b-43df-a4ec-45cae56db52f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6pcxm" Oct 30 23:11:28.308542 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.307313 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae856da3-bc5b-43df-a4ec-45cae56db52f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6pcxm\" (UID: \"ae856da3-bc5b-43df-a4ec-45cae56db52f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6pcxm" Oct 30 23:11:28.308542 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.307728 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae856da3-bc5b-43df-a4ec-45cae56db52f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6pcxm\" (UID: \"ae856da3-bc5b-43df-a4ec-45cae56db52f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6pcxm" Oct 30 23:11:28.308542 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.308357 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae856da3-bc5b-43df-a4ec-45cae56db52f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6pcxm\" (UID: \"ae856da3-bc5b-43df-a4ec-45cae56db52f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6pcxm" Oct 30 23:11:28.361997 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.361934 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:11:28.365500 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.365472 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64mrb\" (UniqueName: \"kubernetes.io/projected/ae856da3-bc5b-43df-a4ec-45cae56db52f-kube-api-access-64mrb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6pcxm\" (UID: \"ae856da3-bc5b-43df-a4ec-45cae56db52f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6pcxm" Oct 30 23:11:28.384913 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.384877 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft" Oct 30 23:11:28.443345 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.443318 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6pcxm" Oct 30 23:11:28.521635 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.521605 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac2glm"] Oct 30 23:11:28.524629 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:11:28.524602 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e1aa4b4_753a_428e_bd78_fd9f2b7b8ab3.slice/crio-2bbbd3c386aa87335d8f04b7420be3ce26b77b13793e42f4b5057e405cf4c657 WatchSource:0}: Error finding container 2bbbd3c386aa87335d8f04b7420be3ce26b77b13793e42f4b5057e405cf4c657: Status 404 returned error can't find the container with id 2bbbd3c386aa87335d8f04b7420be3ce26b77b13793e42f4b5057e405cf4c657 Oct 30 23:11:28.530714 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.530692 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vlm92"] Oct 30 23:11:28.535002 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.534987 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vlm92" Oct 30 23:11:28.590006 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.589974 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-267k7\"" Oct 30 23:11:28.590879 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.590857 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vlm92"] Oct 30 23:11:28.609398 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.609374 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbf82ede-9923-4c1e-a73a-9974c5eca3bc-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vlm92\" (UID: \"fbf82ede-9923-4c1e-a73a-9974c5eca3bc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vlm92" Oct 30 23:11:28.609472 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.609417 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvs64\" (UniqueName: \"kubernetes.io/projected/fbf82ede-9923-4c1e-a73a-9974c5eca3bc-kube-api-access-zvs64\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vlm92\" (UID: \"fbf82ede-9923-4c1e-a73a-9974c5eca3bc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vlm92" Oct 30 23:11:28.609472 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.609443 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbf82ede-9923-4c1e-a73a-9974c5eca3bc-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vlm92\" (UID: \"fbf82ede-9923-4c1e-a73a-9974c5eca3bc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vlm92" Oct 30 23:11:28.627610 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.627561 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78trw2"] Oct 30 23:11:28.630463 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:11:28.630436 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0bbb2ec_d17f_4b9f_8fa0_8affd41bbc41.slice/crio-936580b7e52d1a72468afc09535948877e45973411cab58d14335835f0e42c18 WatchSource:0}: Error finding container 936580b7e52d1a72468afc09535948877e45973411cab58d14335835f0e42c18: Status 404 returned error can't find the container with id 936580b7e52d1a72468afc09535948877e45973411cab58d14335835f0e42c18 Oct 30 23:11:28.710473 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.710449 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbf82ede-9923-4c1e-a73a-9974c5eca3bc-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vlm92\" (UID: \"fbf82ede-9923-4c1e-a73a-9974c5eca3bc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vlm92" Oct 30 23:11:28.710585 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.710502 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbf82ede-9923-4c1e-a73a-9974c5eca3bc-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vlm92\" (UID: \"fbf82ede-9923-4c1e-a73a-9974c5eca3bc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vlm92" Oct 30 23:11:28.710585 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.710547 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvs64\" (UniqueName: \"kubernetes.io/projected/fbf82ede-9923-4c1e-a73a-9974c5eca3bc-kube-api-access-zvs64\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vlm92\" (UID: \"fbf82ede-9923-4c1e-a73a-9974c5eca3bc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vlm92" Oct 30 23:11:28.710803 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.710786 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbf82ede-9923-4c1e-a73a-9974c5eca3bc-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vlm92\" (UID: \"fbf82ede-9923-4c1e-a73a-9974c5eca3bc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vlm92" Oct 30 23:11:28.710900 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.710884 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbf82ede-9923-4c1e-a73a-9974c5eca3bc-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vlm92\" (UID: \"fbf82ede-9923-4c1e-a73a-9974c5eca3bc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vlm92" Oct 30 23:11:28.752514 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.752406 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft"] Oct 30 23:11:28.754398 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:11:28.754372 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91651ce2_4673_4ced_b26d_e94ff8829d11.slice/crio-c9602329f18665a19be7aafa042d3cd75b799e74793c54bc7eca9bd7efcd72f1 WatchSource:0}: Error finding container c9602329f18665a19be7aafa042d3cd75b799e74793c54bc7eca9bd7efcd72f1: Status 404 returned error can't find the container with id c9602329f18665a19be7aafa042d3cd75b799e74793c54bc7eca9bd7efcd72f1 Oct 30 23:11:28.766798 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.766778 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvs64\" (UniqueName: \"kubernetes.io/projected/fbf82ede-9923-4c1e-a73a-9974c5eca3bc-kube-api-access-zvs64\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vlm92\" (UID: \"fbf82ede-9923-4c1e-a73a-9974c5eca3bc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vlm92" Oct 30 23:11:28.810018 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.809978 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6pcxm"] Oct 30 23:11:28.812342 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:11:28.812320 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae856da3_bc5b_43df_a4ec_45cae56db52f.slice/crio-b68365b526fb488d72bd4ad3701910228e1d7963501f67362dc0af36b92c60f6 WatchSource:0}: Error finding container b68365b526fb488d72bd4ad3701910228e1d7963501f67362dc0af36b92c60f6: Status 404 returned error can't find the container with id b68365b526fb488d72bd4ad3701910228e1d7963501f67362dc0af36b92c60f6 Oct 30 23:11:28.850744 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:28.850721 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vlm92" Oct 30 23:11:29.159894 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:29.159811 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78trw2" event={"ID":"b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41","Type":"ContainerStarted","Data":"936580b7e52d1a72468afc09535948877e45973411cab58d14335835f0e42c18"} Oct 30 23:11:29.161013 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:29.160986 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6pcxm" event={"ID":"ae856da3-bc5b-43df-a4ec-45cae56db52f","Type":"ContainerStarted","Data":"b68365b526fb488d72bd4ad3701910228e1d7963501f67362dc0af36b92c60f6"} Oct 30 23:11:29.162154 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:29.162132 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft" event={"ID":"91651ce2-4673-4ced-b26d-e94ff8829d11","Type":"ContainerStarted","Data":"c9602329f18665a19be7aafa042d3cd75b799e74793c54bc7eca9bd7efcd72f1"} Oct 30 23:11:29.163160 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:29.163138 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac2glm" event={"ID":"8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3","Type":"ContainerStarted","Data":"2bbbd3c386aa87335d8f04b7420be3ce26b77b13793e42f4b5057e405cf4c657"} Oct 30 23:11:29.217948 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:29.217758 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vlm92"] Oct 30 23:11:29.220305 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:11:29.220276 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf82ede_9923_4c1e_a73a_9974c5eca3bc.slice/crio-dc0d10bc8098f56e85f16d1a966edc926416585f1a1878a7a5fca8e8de57b388 WatchSource:0}: Error finding container dc0d10bc8098f56e85f16d1a966edc926416585f1a1878a7a5fca8e8de57b388: Status 404 returned error can't find the container with id dc0d10bc8098f56e85f16d1a966edc926416585f1a1878a7a5fca8e8de57b388 Oct 30 23:11:30.174438 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:30.174349 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vlm92" event={"ID":"fbf82ede-9923-4c1e-a73a-9974c5eca3bc","Type":"ContainerStarted","Data":"dc0d10bc8098f56e85f16d1a966edc926416585f1a1878a7a5fca8e8de57b388"} Oct 30 23:11:30.227377 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:30.227314 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nb95d"] Oct 30 23:11:30.237408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:30.237363 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nb95d" Oct 30 23:11:30.288657 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:30.288603 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nb95d"] Oct 30 23:11:30.324626 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:30.324593 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s472n\" (UniqueName: \"kubernetes.io/projected/4291c41e-29e3-49f1-9552-4a417d0bc0d2-kube-api-access-s472n\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nb95d\" (UID: \"4291c41e-29e3-49f1-9552-4a417d0bc0d2\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nb95d" Oct 30 23:11:30.324794 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:30.324674 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4291c41e-29e3-49f1-9552-4a417d0bc0d2-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nb95d\" (UID: \"4291c41e-29e3-49f1-9552-4a417d0bc0d2\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nb95d" Oct 30 23:11:30.324794 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:30.324704 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4291c41e-29e3-49f1-9552-4a417d0bc0d2-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nb95d\" (UID: \"4291c41e-29e3-49f1-9552-4a417d0bc0d2\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nb95d" Oct 30 23:11:30.337180 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:30.337153 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-hwhsn\"" Oct 30 23:11:30.425772 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:30.425675 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4291c41e-29e3-49f1-9552-4a417d0bc0d2-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nb95d\" (UID: \"4291c41e-29e3-49f1-9552-4a417d0bc0d2\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nb95d" Oct 30 23:11:30.425772 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:30.425726 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4291c41e-29e3-49f1-9552-4a417d0bc0d2-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nb95d\" (UID: \"4291c41e-29e3-49f1-9552-4a417d0bc0d2\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nb95d" Oct 30 23:11:30.426009 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:30.425804 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s472n\" (UniqueName: \"kubernetes.io/projected/4291c41e-29e3-49f1-9552-4a417d0bc0d2-kube-api-access-s472n\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nb95d\" (UID: \"4291c41e-29e3-49f1-9552-4a417d0bc0d2\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nb95d" Oct 30 23:11:30.426508 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:30.426475 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4291c41e-29e3-49f1-9552-4a417d0bc0d2-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nb95d\" (UID: \"4291c41e-29e3-49f1-9552-4a417d0bc0d2\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nb95d" Oct 30 23:11:30.426768 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:30.426721 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4291c41e-29e3-49f1-9552-4a417d0bc0d2-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nb95d\" (UID: \"4291c41e-29e3-49f1-9552-4a417d0bc0d2\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nb95d" Oct 30 23:11:30.489971 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:30.489937 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s472n\" (UniqueName: \"kubernetes.io/projected/4291c41e-29e3-49f1-9552-4a417d0bc0d2-kube-api-access-s472n\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nb95d\" (UID: \"4291c41e-29e3-49f1-9552-4a417d0bc0d2\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nb95d" Oct 30 23:11:30.562969 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:30.562932 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nb95d" Oct 30 23:11:30.960744 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:30.958796 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nb95d"] Oct 30 23:11:30.973332 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:11:30.973300 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4291c41e_29e3_49f1_9552_4a417d0bc0d2.slice/crio-a60e98aac11cac0274decd73e940fc63752e4568326c84e42723b60b35c792d3 WatchSource:0}: Error finding container a60e98aac11cac0274decd73e940fc63752e4568326c84e42723b60b35c792d3: Status 404 returned error can't find the container with id a60e98aac11cac0274decd73e940fc63752e4568326c84e42723b60b35c792d3 Oct 30 23:11:30.978442 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:30.978422 4952 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 23:11:31.184201 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:31.184159 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nb95d" event={"ID":"4291c41e-29e3-49f1-9552-4a417d0bc0d2","Type":"ContainerStarted","Data":"a60e98aac11cac0274decd73e940fc63752e4568326c84e42723b60b35c792d3"} Oct 30 23:11:32.539801 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:32.539765 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd7cww"] Oct 30 23:11:32.544427 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:32.544388 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd7cww" Oct 30 23:11:32.597326 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:32.597294 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-nn9z7\"" Oct 30 23:11:32.603022 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:32.602992 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd7cww"] Oct 30 23:11:32.649937 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:32.649897 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a69b88b7-db93-4c88-9763-610ecf0e5590-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd7cww\" (UID: \"a69b88b7-db93-4c88-9763-610ecf0e5590\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd7cww" Oct 30 23:11:32.650135 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:32.649960 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7s5m\" (UniqueName: \"kubernetes.io/projected/a69b88b7-db93-4c88-9763-610ecf0e5590-kube-api-access-d7s5m\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd7cww\" (UID: \"a69b88b7-db93-4c88-9763-610ecf0e5590\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd7cww" Oct 30 23:11:32.650135 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:32.650019 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a69b88b7-db93-4c88-9763-610ecf0e5590-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd7cww\" (UID: \"a69b88b7-db93-4c88-9763-610ecf0e5590\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd7cww" Oct 30 23:11:32.752362 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:32.751153 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a69b88b7-db93-4c88-9763-610ecf0e5590-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd7cww\" (UID: \"a69b88b7-db93-4c88-9763-610ecf0e5590\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd7cww" Oct 30 23:11:32.752362 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:32.751244 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a69b88b7-db93-4c88-9763-610ecf0e5590-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd7cww\" (UID: \"a69b88b7-db93-4c88-9763-610ecf0e5590\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd7cww" Oct 30 23:11:32.752362 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:32.751296 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7s5m\" (UniqueName: \"kubernetes.io/projected/a69b88b7-db93-4c88-9763-610ecf0e5590-kube-api-access-d7s5m\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd7cww\" (UID: \"a69b88b7-db93-4c88-9763-610ecf0e5590\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd7cww" Oct 30 23:11:32.752362 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:32.752314 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a69b88b7-db93-4c88-9763-610ecf0e5590-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd7cww\" (UID: \"a69b88b7-db93-4c88-9763-610ecf0e5590\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd7cww" Oct 30 23:11:32.752879 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:32.752835 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a69b88b7-db93-4c88-9763-610ecf0e5590-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd7cww\" (UID: \"a69b88b7-db93-4c88-9763-610ecf0e5590\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd7cww" Oct 30 23:11:32.813448 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:32.813373 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7s5m\" (UniqueName: \"kubernetes.io/projected/a69b88b7-db93-4c88-9763-610ecf0e5590-kube-api-access-d7s5m\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd7cww\" (UID: \"a69b88b7-db93-4c88-9763-610ecf0e5590\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd7cww" Oct 30 23:11:32.866981 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:32.866947 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd7cww" Oct 30 23:11:33.294251 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:33.292395 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd7cww"] Oct 30 23:11:33.475882 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:33.475844 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:11:33.710499 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:33.710424 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:11:34.206248 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.206138 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd7cww" event={"ID":"a69b88b7-db93-4c88-9763-610ecf0e5590","Type":"ContainerStarted","Data":"d439968e3d8120e4324f5a7b7ffdf425288670c1ef06248e901b7b684c0a9ad3"} Oct 30 23:11:34.359088 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.357781 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x"] Oct 30 23:11:34.363310 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.362432 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:11:34.418995 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.418822 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 23:11:34.438625 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.438600 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-l8k5z\"" Oct 30 23:11:34.470794 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.470718 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x"] Oct 30 23:11:34.473287 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.473257 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbpfj\" (UniqueName: \"kubernetes.io/projected/f5421fcf-649c-477a-b487-68c52d23f09b-kube-api-access-pbpfj\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dz26x\" (UID: \"f5421fcf-649c-477a-b487-68c52d23f09b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:11:34.473411 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.473312 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5421fcf-649c-477a-b487-68c52d23f09b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dz26x\" (UID: \"f5421fcf-649c-477a-b487-68c52d23f09b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:11:34.473411 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.473373 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5421fcf-649c-477a-b487-68c52d23f09b-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dz26x\" (UID: \"f5421fcf-649c-477a-b487-68c52d23f09b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:11:34.473411 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.473400 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5421fcf-649c-477a-b487-68c52d23f09b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dz26x\" (UID: \"f5421fcf-649c-477a-b487-68c52d23f09b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:11:34.473575 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.473446 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5421fcf-649c-477a-b487-68c52d23f09b-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dz26x\" (UID: \"f5421fcf-649c-477a-b487-68c52d23f09b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:11:34.473575 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.473483 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5421fcf-649c-477a-b487-68c52d23f09b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dz26x\" (UID: \"f5421fcf-649c-477a-b487-68c52d23f09b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:11:34.473575 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.473522 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5421fcf-649c-477a-b487-68c52d23f09b-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dz26x\" (UID: \"f5421fcf-649c-477a-b487-68c52d23f09b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:11:34.473575 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.473546 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5421fcf-649c-477a-b487-68c52d23f09b-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dz26x\" (UID: \"f5421fcf-649c-477a-b487-68c52d23f09b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:11:34.473772 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.473611 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5421fcf-649c-477a-b487-68c52d23f09b-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dz26x\" (UID: \"f5421fcf-649c-477a-b487-68c52d23f09b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:11:34.543832 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.543696 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:11:34.575420 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.574669 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5421fcf-649c-477a-b487-68c52d23f09b-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dz26x\" (UID: \"f5421fcf-649c-477a-b487-68c52d23f09b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:11:34.575420 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.574701 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5421fcf-649c-477a-b487-68c52d23f09b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dz26x\" (UID: \"f5421fcf-649c-477a-b487-68c52d23f09b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:11:34.575420 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.574728 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5421fcf-649c-477a-b487-68c52d23f09b-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dz26x\" (UID: \"f5421fcf-649c-477a-b487-68c52d23f09b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:11:34.575420 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.574753 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5421fcf-649c-477a-b487-68c52d23f09b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dz26x\" (UID: \"f5421fcf-649c-477a-b487-68c52d23f09b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:11:34.575420 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.574773 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5421fcf-649c-477a-b487-68c52d23f09b-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dz26x\" (UID: \"f5421fcf-649c-477a-b487-68c52d23f09b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:11:34.575420 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.574790 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5421fcf-649c-477a-b487-68c52d23f09b-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dz26x\" (UID: \"f5421fcf-649c-477a-b487-68c52d23f09b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:11:34.575420 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.574825 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5421fcf-649c-477a-b487-68c52d23f09b-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dz26x\" (UID: \"f5421fcf-649c-477a-b487-68c52d23f09b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:11:34.575420 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.574855 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbpfj\" (UniqueName: \"kubernetes.io/projected/f5421fcf-649c-477a-b487-68c52d23f09b-kube-api-access-pbpfj\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dz26x\" (UID: \"f5421fcf-649c-477a-b487-68c52d23f09b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:11:34.575420 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.574874 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5421fcf-649c-477a-b487-68c52d23f09b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dz26x\" (UID: \"f5421fcf-649c-477a-b487-68c52d23f09b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:11:34.578181 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.576814 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5421fcf-649c-477a-b487-68c52d23f09b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dz26x\" (UID: \"f5421fcf-649c-477a-b487-68c52d23f09b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:11:34.578181 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.577529 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5421fcf-649c-477a-b487-68c52d23f09b-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dz26x\" (UID: \"f5421fcf-649c-477a-b487-68c52d23f09b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:11:34.578181 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.577628 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5421fcf-649c-477a-b487-68c52d23f09b-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dz26x\" (UID: \"f5421fcf-649c-477a-b487-68c52d23f09b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:11:34.578181 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.577811 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5421fcf-649c-477a-b487-68c52d23f09b-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dz26x\" (UID: \"f5421fcf-649c-477a-b487-68c52d23f09b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:11:34.578181 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.577906 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5421fcf-649c-477a-b487-68c52d23f09b-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dz26x\" (UID: \"f5421fcf-649c-477a-b487-68c52d23f09b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:11:34.582395 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.582233 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5421fcf-649c-477a-b487-68c52d23f09b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dz26x\" (UID: \"f5421fcf-649c-477a-b487-68c52d23f09b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:11:34.585578 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.585274 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5421fcf-649c-477a-b487-68c52d23f09b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dz26x\" (UID: \"f5421fcf-649c-477a-b487-68c52d23f09b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:11:34.648929 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.648844 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5421fcf-649c-477a-b487-68c52d23f09b-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dz26x\" (UID: \"f5421fcf-649c-477a-b487-68c52d23f09b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:11:34.653385 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.653356 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbpfj\" (UniqueName: \"kubernetes.io/projected/f5421fcf-649c-477a-b487-68c52d23f09b-kube-api-access-pbpfj\") pod \"maas-default-gateway-openshift-default-5df48fcc99-dz26x\" (UID: \"f5421fcf-649c-477a-b487-68c52d23f09b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:11:34.680708 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.680578 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:11:34.710485 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.710455 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct"] Oct 30 23:11:34.715786 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.714955 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" Oct 30 23:11:34.767757 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.767732 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-qxtht\"" Oct 30 23:11:34.778018 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.777948 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct"] Oct 30 23:11:34.780272 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.779031 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82aebc2-0acd-4643-9481-7f1c0c6ddf35-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct\" (UID: \"a82aebc2-0acd-4643-9481-7f1c0c6ddf35\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" Oct 30 23:11:34.780272 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.779151 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82aebc2-0acd-4643-9481-7f1c0c6ddf35-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct\" (UID: \"a82aebc2-0acd-4643-9481-7f1c0c6ddf35\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" Oct 30 23:11:34.780272 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.779189 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a82aebc2-0acd-4643-9481-7f1c0c6ddf35-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct\" (UID: \"a82aebc2-0acd-4643-9481-7f1c0c6ddf35\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" Oct 30 23:11:34.780272 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.779215 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a82aebc2-0acd-4643-9481-7f1c0c6ddf35-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct\" (UID: \"a82aebc2-0acd-4643-9481-7f1c0c6ddf35\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" Oct 30 23:11:34.780272 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.779235 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a82aebc2-0acd-4643-9481-7f1c0c6ddf35-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct\" (UID: \"a82aebc2-0acd-4643-9481-7f1c0c6ddf35\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" Oct 30 23:11:34.780272 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.779274 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a82aebc2-0acd-4643-9481-7f1c0c6ddf35-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct\" (UID: \"a82aebc2-0acd-4643-9481-7f1c0c6ddf35\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" Oct 30 23:11:34.780272 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.779342 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a82aebc2-0acd-4643-9481-7f1c0c6ddf35-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct\" (UID: \"a82aebc2-0acd-4643-9481-7f1c0c6ddf35\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" Oct 30 23:11:34.780272 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.779368 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a82aebc2-0acd-4643-9481-7f1c0c6ddf35-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct\" (UID: \"a82aebc2-0acd-4643-9481-7f1c0c6ddf35\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" Oct 30 23:11:34.780272 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.779398 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c786k\" (UniqueName: \"kubernetes.io/projected/a82aebc2-0acd-4643-9481-7f1c0c6ddf35-kube-api-access-c786k\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct\" (UID: \"a82aebc2-0acd-4643-9481-7f1c0c6ddf35\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" Oct 30 23:11:34.878297 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.878241 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:11:34.881463 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.879915 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a82aebc2-0acd-4643-9481-7f1c0c6ddf35-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct\" (UID: \"a82aebc2-0acd-4643-9481-7f1c0c6ddf35\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" Oct 30 23:11:34.881463 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.879991 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a82aebc2-0acd-4643-9481-7f1c0c6ddf35-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct\" (UID: \"a82aebc2-0acd-4643-9481-7f1c0c6ddf35\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" Oct 30 23:11:34.881463 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.880017 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a82aebc2-0acd-4643-9481-7f1c0c6ddf35-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct\" (UID: \"a82aebc2-0acd-4643-9481-7f1c0c6ddf35\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" Oct 30 23:11:34.881463 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.880069 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c786k\" (UniqueName: \"kubernetes.io/projected/a82aebc2-0acd-4643-9481-7f1c0c6ddf35-kube-api-access-c786k\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct\" (UID: \"a82aebc2-0acd-4643-9481-7f1c0c6ddf35\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" Oct 30 23:11:34.881463 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.880132 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82aebc2-0acd-4643-9481-7f1c0c6ddf35-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct\" (UID: \"a82aebc2-0acd-4643-9481-7f1c0c6ddf35\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" Oct 30 23:11:34.881463 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.880168 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82aebc2-0acd-4643-9481-7f1c0c6ddf35-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct\" (UID: \"a82aebc2-0acd-4643-9481-7f1c0c6ddf35\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" Oct 30 23:11:34.881463 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.880196 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a82aebc2-0acd-4643-9481-7f1c0c6ddf35-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct\" (UID: \"a82aebc2-0acd-4643-9481-7f1c0c6ddf35\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" Oct 30 23:11:34.881463 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.880230 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a82aebc2-0acd-4643-9481-7f1c0c6ddf35-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct\" (UID: \"a82aebc2-0acd-4643-9481-7f1c0c6ddf35\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" Oct 30 23:11:34.881463 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.880258 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a82aebc2-0acd-4643-9481-7f1c0c6ddf35-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct\" (UID: \"a82aebc2-0acd-4643-9481-7f1c0c6ddf35\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" Oct 30 23:11:34.881463 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.880671 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a82aebc2-0acd-4643-9481-7f1c0c6ddf35-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct\" (UID: \"a82aebc2-0acd-4643-9481-7f1c0c6ddf35\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" Oct 30 23:11:34.884291 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.882417 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82aebc2-0acd-4643-9481-7f1c0c6ddf35-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct\" (UID: \"a82aebc2-0acd-4643-9481-7f1c0c6ddf35\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" Oct 30 23:11:34.884291 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.883407 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a82aebc2-0acd-4643-9481-7f1c0c6ddf35-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct\" (UID: \"a82aebc2-0acd-4643-9481-7f1c0c6ddf35\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" Oct 30 23:11:34.884291 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.883647 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82aebc2-0acd-4643-9481-7f1c0c6ddf35-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct\" (UID: \"a82aebc2-0acd-4643-9481-7f1c0c6ddf35\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" Oct 30 23:11:34.884291 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.884242 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a82aebc2-0acd-4643-9481-7f1c0c6ddf35-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct\" (UID: \"a82aebc2-0acd-4643-9481-7f1c0c6ddf35\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" Oct 30 23:11:34.885738 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.885673 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a82aebc2-0acd-4643-9481-7f1c0c6ddf35-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct\" (UID: \"a82aebc2-0acd-4643-9481-7f1c0c6ddf35\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" Oct 30 23:11:34.888568 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.888539 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a82aebc2-0acd-4643-9481-7f1c0c6ddf35-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct\" (UID: \"a82aebc2-0acd-4643-9481-7f1c0c6ddf35\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" Oct 30 23:11:34.940858 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.940416 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a82aebc2-0acd-4643-9481-7f1c0c6ddf35-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct\" (UID: \"a82aebc2-0acd-4643-9481-7f1c0c6ddf35\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" Oct 30 23:11:34.941129 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:34.941094 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c786k\" (UniqueName: \"kubernetes.io/projected/a82aebc2-0acd-4643-9481-7f1c0c6ddf35-kube-api-access-c786k\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct\" (UID: \"a82aebc2-0acd-4643-9481-7f1c0c6ddf35\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" Oct 30 23:11:35.037154 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.036646 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" Oct 30 23:11:35.112679 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.112548 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:11:35.165740 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.165598 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x"] Oct 30 23:11:35.218529 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.218389 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" event={"ID":"f5421fcf-649c-477a-b487-68c52d23f09b","Type":"ContainerStarted","Data":"c85f523fdc56b8924e9d71d5f643f9f5957e4d36cc155a22904886dc16a38d83"} Oct 30 23:11:35.444877 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.444841 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tg9xk"] Oct 30 23:11:35.451221 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.450587 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tg9xk" Oct 30 23:11:35.486653 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.484886 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lbh9v"] Oct 30 23:11:35.489291 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.488838 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lbh9v" Oct 30 23:11:35.506902 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.506882 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct"] Oct 30 23:11:35.507009 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.506939 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tg9xk"] Oct 30 23:11:35.533302 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.531283 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz7h28"] Oct 30 23:11:35.535919 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.535543 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz7h28" Oct 30 23:11:35.547490 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.547466 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lbh9v"] Oct 30 23:11:35.579455 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.579374 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar4k5p"] Oct 30 23:11:35.586081 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.584424 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar4k5p" Oct 30 23:11:35.586451 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.586421 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/91fede71-5a6e-40fa-873b-16c20fedb00c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lbh9v\" (UID: \"91fede71-5a6e-40fa-873b-16c20fedb00c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lbh9v" Oct 30 23:11:35.586451 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.586479 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsqzq\" (UniqueName: \"kubernetes.io/projected/1166c90f-c4d5-4e7d-abd7-bde3f2c44010-kube-api-access-xsqzq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tg9xk\" (UID: \"1166c90f-c4d5-4e7d-abd7-bde3f2c44010\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tg9xk" Oct 30 23:11:35.586744 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.586517 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6222\" (UniqueName: \"kubernetes.io/projected/91fede71-5a6e-40fa-873b-16c20fedb00c-kube-api-access-l6222\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lbh9v\" (UID: \"91fede71-5a6e-40fa-873b-16c20fedb00c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lbh9v" Oct 30 23:11:35.586744 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.586563 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz7h28\" (UID: \"841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz7h28" Oct 30 23:11:35.586744 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.586631 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcxd4\" (UniqueName: \"kubernetes.io/projected/841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1-kube-api-access-tcxd4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz7h28\" (UID: \"841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz7h28" Oct 30 23:11:35.586744 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.586674 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1166c90f-c4d5-4e7d-abd7-bde3f2c44010-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tg9xk\" (UID: \"1166c90f-c4d5-4e7d-abd7-bde3f2c44010\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tg9xk" Oct 30 23:11:35.586975 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.586747 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1166c90f-c4d5-4e7d-abd7-bde3f2c44010-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tg9xk\" (UID: \"1166c90f-c4d5-4e7d-abd7-bde3f2c44010\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tg9xk" Oct 30 23:11:35.586975 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.586783 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/91fede71-5a6e-40fa-873b-16c20fedb00c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lbh9v\" (UID: \"91fede71-5a6e-40fa-873b-16c20fedb00c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lbh9v" Oct 30 23:11:35.586975 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.586822 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz7h28\" (UID: \"841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz7h28" Oct 30 23:11:35.613785 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.613758 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz7h28"] Oct 30 23:11:35.642185 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.640789 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar4k5p"] Oct 30 23:11:35.687683 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.687647 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1166c90f-c4d5-4e7d-abd7-bde3f2c44010-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tg9xk\" (UID: \"1166c90f-c4d5-4e7d-abd7-bde3f2c44010\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tg9xk" Oct 30 23:11:35.687683 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.687699 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcvvm\" (UniqueName: \"kubernetes.io/projected/e883e62d-ee03-411c-8fac-e9dacb0dc5e9-kube-api-access-lcvvm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar4k5p\" (UID: \"e883e62d-ee03-411c-8fac-e9dacb0dc5e9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar4k5p" Oct 30 23:11:35.687940 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.687745 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/91fede71-5a6e-40fa-873b-16c20fedb00c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lbh9v\" (UID: \"91fede71-5a6e-40fa-873b-16c20fedb00c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lbh9v" Oct 30 23:11:35.687940 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.687784 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz7h28\" (UID: \"841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz7h28" Oct 30 23:11:35.687940 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.687821 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/91fede71-5a6e-40fa-873b-16c20fedb00c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lbh9v\" (UID: \"91fede71-5a6e-40fa-873b-16c20fedb00c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lbh9v" Oct 30 23:11:35.687940 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.687855 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e883e62d-ee03-411c-8fac-e9dacb0dc5e9-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar4k5p\" (UID: \"e883e62d-ee03-411c-8fac-e9dacb0dc5e9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar4k5p" Oct 30 23:11:35.687940 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.687888 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsqzq\" (UniqueName: \"kubernetes.io/projected/1166c90f-c4d5-4e7d-abd7-bde3f2c44010-kube-api-access-xsqzq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tg9xk\" (UID: \"1166c90f-c4d5-4e7d-abd7-bde3f2c44010\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tg9xk" Oct 30 23:11:35.687940 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.687916 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6222\" (UniqueName: \"kubernetes.io/projected/91fede71-5a6e-40fa-873b-16c20fedb00c-kube-api-access-l6222\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lbh9v\" (UID: \"91fede71-5a6e-40fa-873b-16c20fedb00c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lbh9v" Oct 30 23:11:35.688457 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.687947 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz7h28\" (UID: \"841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz7h28" Oct 30 23:11:35.688457 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.687995 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcxd4\" (UniqueName: \"kubernetes.io/projected/841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1-kube-api-access-tcxd4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz7h28\" (UID: \"841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz7h28" Oct 30 23:11:35.688457 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.688053 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1166c90f-c4d5-4e7d-abd7-bde3f2c44010-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tg9xk\" (UID: \"1166c90f-c4d5-4e7d-abd7-bde3f2c44010\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tg9xk" Oct 30 23:11:35.688457 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.688082 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e883e62d-ee03-411c-8fac-e9dacb0dc5e9-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar4k5p\" (UID: \"e883e62d-ee03-411c-8fac-e9dacb0dc5e9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar4k5p" Oct 30 23:11:35.688457 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.688130 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1166c90f-c4d5-4e7d-abd7-bde3f2c44010-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tg9xk\" (UID: \"1166c90f-c4d5-4e7d-abd7-bde3f2c44010\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tg9xk" Oct 30 23:11:35.688779 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.688487 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1166c90f-c4d5-4e7d-abd7-bde3f2c44010-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tg9xk\" (UID: \"1166c90f-c4d5-4e7d-abd7-bde3f2c44010\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tg9xk" Oct 30 23:11:35.688779 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.688530 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/91fede71-5a6e-40fa-873b-16c20fedb00c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lbh9v\" (UID: \"91fede71-5a6e-40fa-873b-16c20fedb00c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lbh9v" Oct 30 23:11:35.688942 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.688838 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz7h28\" (UID: \"841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz7h28" Oct 30 23:11:35.688942 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.688863 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz7h28\" (UID: \"841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz7h28" Oct 30 23:11:35.689421 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.689396 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/91fede71-5a6e-40fa-873b-16c20fedb00c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lbh9v\" (UID: \"91fede71-5a6e-40fa-873b-16c20fedb00c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lbh9v" Oct 30 23:11:35.747608 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.747535 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsqzq\" (UniqueName: \"kubernetes.io/projected/1166c90f-c4d5-4e7d-abd7-bde3f2c44010-kube-api-access-xsqzq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tg9xk\" (UID: \"1166c90f-c4d5-4e7d-abd7-bde3f2c44010\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tg9xk" Oct 30 23:11:35.751399 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.750975 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcxd4\" (UniqueName: \"kubernetes.io/projected/841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1-kube-api-access-tcxd4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz7h28\" (UID: \"841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz7h28" Oct 30 23:11:35.751481 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.751445 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6222\" (UniqueName: \"kubernetes.io/projected/91fede71-5a6e-40fa-873b-16c20fedb00c-kube-api-access-l6222\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lbh9v\" (UID: \"91fede71-5a6e-40fa-873b-16c20fedb00c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lbh9v" Oct 30 23:11:35.772680 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.772462 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tg9xk" Oct 30 23:11:35.789161 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.789138 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcvvm\" (UniqueName: \"kubernetes.io/projected/e883e62d-ee03-411c-8fac-e9dacb0dc5e9-kube-api-access-lcvvm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar4k5p\" (UID: \"e883e62d-ee03-411c-8fac-e9dacb0dc5e9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar4k5p" Oct 30 23:11:35.789271 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.789191 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e883e62d-ee03-411c-8fac-e9dacb0dc5e9-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar4k5p\" (UID: \"e883e62d-ee03-411c-8fac-e9dacb0dc5e9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar4k5p" Oct 30 23:11:35.789271 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.789250 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e883e62d-ee03-411c-8fac-e9dacb0dc5e9-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar4k5p\" (UID: \"e883e62d-ee03-411c-8fac-e9dacb0dc5e9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar4k5p" Oct 30 23:11:35.789794 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.789551 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e883e62d-ee03-411c-8fac-e9dacb0dc5e9-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar4k5p\" (UID: \"e883e62d-ee03-411c-8fac-e9dacb0dc5e9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar4k5p" Oct 30 23:11:35.789794 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.789743 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e883e62d-ee03-411c-8fac-e9dacb0dc5e9-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar4k5p\" (UID: \"e883e62d-ee03-411c-8fac-e9dacb0dc5e9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar4k5p" Oct 30 23:11:35.813457 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.813240 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lbh9v" Oct 30 23:11:35.849704 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.849674 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcvvm\" (UniqueName: \"kubernetes.io/projected/e883e62d-ee03-411c-8fac-e9dacb0dc5e9-kube-api-access-lcvvm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar4k5p\" (UID: \"e883e62d-ee03-411c-8fac-e9dacb0dc5e9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar4k5p" Oct 30 23:11:35.873695 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.870563 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz7h28" Oct 30 23:11:35.968538 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:35.968299 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar4k5p" Oct 30 23:11:36.179963 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:36.179820 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tg9xk"] Oct 30 23:11:36.226959 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:36.226894 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" event={"ID":"a82aebc2-0acd-4643-9481-7f1c0c6ddf35","Type":"ContainerStarted","Data":"a5171a7193b0b27deaf8d40dd93d51eaed7f6705548d9487acdf26ca61e1b6eb"} Oct 30 23:11:36.227349 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:36.227326 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lbh9v"] Oct 30 23:11:36.232511 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:11:36.232323 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91fede71_5a6e_40fa_873b_16c20fedb00c.slice/crio-07dde687075c97ffcb4a062ad5b51e7adb0780358b488447c46f4281305b910b WatchSource:0}: Error finding container 07dde687075c97ffcb4a062ad5b51e7adb0780358b488447c46f4281305b910b: Status 404 returned error can't find the container with id 07dde687075c97ffcb4a062ad5b51e7adb0780358b488447c46f4281305b910b Oct 30 23:11:36.236116 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:36.236088 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tg9xk" event={"ID":"1166c90f-c4d5-4e7d-abd7-bde3f2c44010","Type":"ContainerStarted","Data":"a7c896ce476d4f962911a68f82a869dea8e26d1b6a1b707b1cd732caff939e48"} Oct 30 23:11:36.350726 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:36.350668 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz7h28"] Oct 30 23:11:36.357152 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:11:36.357112 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod841bd0a8_32f9_4ea8_afe4_1a8127a5a1f1.slice/crio-fb23622cc446e2aef00b392e0171ee9edd80c5cd4c4edcd1c17d8076307cd5c4 WatchSource:0}: Error finding container fb23622cc446e2aef00b392e0171ee9edd80c5cd4c4edcd1c17d8076307cd5c4: Status 404 returned error can't find the container with id fb23622cc446e2aef00b392e0171ee9edd80c5cd4c4edcd1c17d8076307cd5c4 Oct 30 23:11:36.398432 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:36.396990 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar4k5p"] Oct 30 23:11:36.399224 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:11:36.399192 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode883e62d_ee03_411c_8fac_e9dacb0dc5e9.slice/crio-e9e88b723e964362c2539832911ef31e441bf0cfd51a813d9efe0c33fe1dd27b WatchSource:0}: Error finding container e9e88b723e964362c2539832911ef31e441bf0cfd51a813d9efe0c33fe1dd27b: Status 404 returned error can't find the container with id e9e88b723e964362c2539832911ef31e441bf0cfd51a813d9efe0c33fe1dd27b Oct 30 23:11:37.253460 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:37.253416 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar4k5p" event={"ID":"e883e62d-ee03-411c-8fac-e9dacb0dc5e9","Type":"ContainerStarted","Data":"e9e88b723e964362c2539832911ef31e441bf0cfd51a813d9efe0c33fe1dd27b"} Oct 30 23:11:37.264260 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:37.264225 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz7h28" event={"ID":"841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1","Type":"ContainerStarted","Data":"fb23622cc446e2aef00b392e0171ee9edd80c5cd4c4edcd1c17d8076307cd5c4"} Oct 30 23:11:37.275017 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:37.274951 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lbh9v" event={"ID":"91fede71-5a6e-40fa-873b-16c20fedb00c","Type":"ContainerStarted","Data":"07dde687075c97ffcb4a062ad5b51e7adb0780358b488447c46f4281305b910b"} Oct 30 23:11:38.783507 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:38.783421 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:11:49.097064 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:49.096737 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:11:49.097064 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:49.096815 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:11:49.097064 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:49.096859 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:11:49.115613 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:49.114872 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:11:49.115613 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:49.115019 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:11:49.115613 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:49.115137 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:11:49.442084 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:49.441155 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" event={"ID":"a82aebc2-0acd-4643-9481-7f1c0c6ddf35","Type":"ContainerStarted","Data":"9075295dcfbc931e6205e0f85e72534ede2d6d04b9ef84232b81ee491671b651"} Oct 30 23:11:49.448894 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:49.448347 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" event={"ID":"f5421fcf-649c-477a-b487-68c52d23f09b","Type":"ContainerStarted","Data":"28fa26b49afd6429443af96350c60890fd9066ae78f90914ebc413ab4f06a79a"} Oct 30 23:11:49.455271 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:49.454525 4952 generic.go:356] "Generic (PLEG): container finished" podID="ae856da3-bc5b-43df-a4ec-45cae56db52f" containerID="a16c77a5ec14d3f54799ab548cedc9c1d702a04630f367d8eaa570c53ce475d4" exitCode=0 Oct 30 23:11:49.455271 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:49.454587 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6pcxm" event={"ID":"ae856da3-bc5b-43df-a4ec-45cae56db52f","Type":"ContainerDied","Data":"a16c77a5ec14d3f54799ab548cedc9c1d702a04630f367d8eaa570c53ce475d4"} Oct 30 23:11:49.600174 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:49.599896 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" podStartSLOduration=2.019325433 podStartE2EDuration="15.599875012s" totalImagesPullingTime="13.580549579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:11:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:11:49.597757296 +0000 UTC m=+328.741587525" watchObservedRunningTime="2026-10-30 23:11:49.599875012 +0000 UTC m=+328.743705220" Oct 30 23:11:49.681640 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:49.681420 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:11:49.681640 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:49.681528 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:11:49.780051 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:11:49.778555 4952 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf82ede_9923_4c1e_a73a_9974c5eca3bc.slice/crio-conmon-722c503c935f4478ffae09603b05329db8e66371ed429ae0ead965ee99c64a6c.scope\": RecentStats: unable to find data in memory cache]" Oct 30 23:11:49.832836 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:49.832628 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" podStartSLOduration=1.897174779 podStartE2EDuration="15.832609851s" totalImagesPullingTime="13.935435072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:11:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:11:49.831870334 +0000 UTC m=+328.975700540" watchObservedRunningTime="2026-10-30 23:11:49.832609851 +0000 UTC m=+328.976440056" Oct 30 23:11:50.038016 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:50.037822 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" Oct 30 23:11:50.038016 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:50.037898 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" Oct 30 23:11:50.044320 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:50.044080 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" Oct 30 23:11:50.461871 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:50.461832 4952 generic.go:356] "Generic (PLEG): container finished" podID="fbf82ede-9923-4c1e-a73a-9974c5eca3bc" containerID="722c503c935f4478ffae09603b05329db8e66371ed429ae0ead965ee99c64a6c" exitCode=0 Oct 30 23:11:50.462343 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:50.461926 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vlm92" event={"ID":"fbf82ede-9923-4c1e-a73a-9974c5eca3bc","Type":"ContainerDied","Data":"722c503c935f4478ffae09603b05329db8e66371ed429ae0ead965ee99c64a6c"} Oct 30 23:11:50.464115 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:50.463991 4952 generic.go:356] "Generic (PLEG): container finished" podID="e883e62d-ee03-411c-8fac-e9dacb0dc5e9" containerID="487cc1dc04f60fbea766b6286fe68c4f6576b31a934227c5e0db3db27596bf26" exitCode=0 Oct 30 23:11:50.464115 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:50.464068 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar4k5p" event={"ID":"e883e62d-ee03-411c-8fac-e9dacb0dc5e9","Type":"ContainerDied","Data":"487cc1dc04f60fbea766b6286fe68c4f6576b31a934227c5e0db3db27596bf26"} Oct 30 23:11:50.466109 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:50.466090 4952 generic.go:356] "Generic (PLEG): container finished" podID="b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41" containerID="4a5546c5c3ea3f865e2bb929b1ff3a483380adb075ef6413bb118a19551205b6" exitCode=0 Oct 30 23:11:50.466211 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:50.466182 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78trw2" event={"ID":"b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41","Type":"ContainerDied","Data":"4a5546c5c3ea3f865e2bb929b1ff3a483380adb075ef6413bb118a19551205b6"} Oct 30 23:11:50.468656 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:50.468551 4952 generic.go:356] "Generic (PLEG): container finished" podID="841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1" containerID="25459b17c06ed9a99d069a3fa14a924338f70c286859b21c48b35f538afb7bd2" exitCode=0 Oct 30 23:11:50.468656 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:50.468623 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz7h28" event={"ID":"841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1","Type":"ContainerDied","Data":"25459b17c06ed9a99d069a3fa14a924338f70c286859b21c48b35f538afb7bd2"} Oct 30 23:11:50.470342 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:50.470322 4952 generic.go:356] "Generic (PLEG): container finished" podID="91fede71-5a6e-40fa-873b-16c20fedb00c" containerID="8f2a2723b71aaacd1971adcb90c87bb75f233fc2a3876a5c82b58e2433efde15" exitCode=0 Oct 30 23:11:50.470568 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:50.470543 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lbh9v" event={"ID":"91fede71-5a6e-40fa-873b-16c20fedb00c","Type":"ContainerDied","Data":"8f2a2723b71aaacd1971adcb90c87bb75f233fc2a3876a5c82b58e2433efde15"} Oct 30 23:11:50.472613 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:50.472590 4952 generic.go:356] "Generic (PLEG): container finished" podID="1166c90f-c4d5-4e7d-abd7-bde3f2c44010" containerID="32440453a2ead926b5d37bdbfff0eed6e8330687e0d65155ff5a0cecf6b285a5" exitCode=0 Oct 30 23:11:50.472719 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:50.472678 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tg9xk" event={"ID":"1166c90f-c4d5-4e7d-abd7-bde3f2c44010","Type":"ContainerDied","Data":"32440453a2ead926b5d37bdbfff0eed6e8330687e0d65155ff5a0cecf6b285a5"} Oct 30 23:11:50.474570 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:50.474480 4952 generic.go:356] "Generic (PLEG): container finished" podID="91651ce2-4673-4ced-b26d-e94ff8829d11" containerID="0b95c53e559852265a8769b72000161032546757fb1a24f7d75b9b6f1d566bda" exitCode=0 Oct 30 23:11:50.474570 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:50.474548 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft" event={"ID":"91651ce2-4673-4ced-b26d-e94ff8829d11","Type":"ContainerDied","Data":"0b95c53e559852265a8769b72000161032546757fb1a24f7d75b9b6f1d566bda"} Oct 30 23:11:50.477851 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:50.477719 4952 generic.go:356] "Generic (PLEG): container finished" podID="8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3" containerID="e8b4d57f3dc0a6761b3c9b5fffe17886ad8157e6026b2a3d84a47c7db0d8b443" exitCode=0 Oct 30 23:11:50.478118 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:50.478094 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac2glm" event={"ID":"8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3","Type":"ContainerDied","Data":"e8b4d57f3dc0a6761b3c9b5fffe17886ad8157e6026b2a3d84a47c7db0d8b443"} Oct 30 23:11:50.479720 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:50.479704 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct" Oct 30 23:11:50.687451 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:50.687390 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:11:51.147306 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:11:51.147245 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:11:51.485281 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:51.485235 4952 generic.go:356] "Generic (PLEG): container finished" podID="4291c41e-29e3-49f1-9552-4a417d0bc0d2" containerID="788ceeb0c61610d6dcedf62730c2bf6c544ef1467bb3f56693587e649c974571" exitCode=0 Oct 30 23:11:51.485746 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:51.485443 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nb95d" event={"ID":"4291c41e-29e3-49f1-9552-4a417d0bc0d2","Type":"ContainerDied","Data":"788ceeb0c61610d6dcedf62730c2bf6c544ef1467bb3f56693587e649c974571"} Oct 30 23:11:51.487123 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:51.487009 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:11:52.492782 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:52.492743 4952 generic.go:356] "Generic (PLEG): container finished" podID="a69b88b7-db93-4c88-9763-610ecf0e5590" containerID="354a756878e4c74e851acd88dcb268efac28a9578f793cbdfed80a9fc09e1ada" exitCode=0 Oct 30 23:11:52.493205 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:52.492837 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd7cww" event={"ID":"a69b88b7-db93-4c88-9763-610ecf0e5590","Type":"ContainerDied","Data":"354a756878e4c74e851acd88dcb268efac28a9578f793cbdfed80a9fc09e1ada"} Oct 30 23:11:52.495883 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:52.495855 4952 generic.go:356] "Generic (PLEG): container finished" podID="e883e62d-ee03-411c-8fac-e9dacb0dc5e9" containerID="12674cd8e3bc44940a65bf32e334c1c4eee8665040acbdf50259c583ea3f4393" exitCode=0 Oct 30 23:11:52.496000 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:52.495938 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar4k5p" event={"ID":"e883e62d-ee03-411c-8fac-e9dacb0dc5e9","Type":"ContainerDied","Data":"12674cd8e3bc44940a65bf32e334c1c4eee8665040acbdf50259c583ea3f4393"} Oct 30 23:11:52.498972 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:52.498948 4952 generic.go:356] "Generic (PLEG): container finished" podID="b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41" containerID="87c1cb51a779bb6f55556106d67a81cff98a21d9e5c296cd4dd12f3ef72a4ae3" exitCode=0 Oct 30 23:11:52.499888 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:52.499861 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78trw2" event={"ID":"b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41","Type":"ContainerDied","Data":"87c1cb51a779bb6f55556106d67a81cff98a21d9e5c296cd4dd12f3ef72a4ae3"} Oct 30 23:11:52.503268 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:52.502793 4952 generic.go:356] "Generic (PLEG): container finished" podID="841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1" containerID="ce173dd071d67e0615f286cd961f1fe5794e1d27701963cd462e53a7cb74aee4" exitCode=0 Oct 30 23:11:52.503268 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:52.502893 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz7h28" event={"ID":"841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1","Type":"ContainerDied","Data":"ce173dd071d67e0615f286cd961f1fe5794e1d27701963cd462e53a7cb74aee4"} Oct 30 23:11:52.508174 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:52.508079 4952 generic.go:356] "Generic (PLEG): container finished" podID="91fede71-5a6e-40fa-873b-16c20fedb00c" containerID="5e9852e75f4b97e3a964cb045f26e3084447cec5bf2fceba47a0b7714e0a7e04" exitCode=0 Oct 30 23:11:52.508174 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:52.508119 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lbh9v" event={"ID":"91fede71-5a6e-40fa-873b-16c20fedb00c","Type":"ContainerDied","Data":"5e9852e75f4b97e3a964cb045f26e3084447cec5bf2fceba47a0b7714e0a7e04"} Oct 30 23:11:52.510451 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:52.510428 4952 generic.go:356] "Generic (PLEG): container finished" podID="1166c90f-c4d5-4e7d-abd7-bde3f2c44010" containerID="df2ee7a10e3371c49f5a3803ded84eab726920361caeb0d3415cfd1c85348920" exitCode=0 Oct 30 23:11:52.510562 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:52.510496 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tg9xk" event={"ID":"1166c90f-c4d5-4e7d-abd7-bde3f2c44010","Type":"ContainerDied","Data":"df2ee7a10e3371c49f5a3803ded84eab726920361caeb0d3415cfd1c85348920"} Oct 30 23:11:52.512574 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:52.512554 4952 generic.go:356] "Generic (PLEG): container finished" podID="ae856da3-bc5b-43df-a4ec-45cae56db52f" containerID="096627295a641646fbd295b19e87ccf9f4a0c71d64af3d14a8641d06bf5df19a" exitCode=0 Oct 30 23:11:52.512701 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:52.512611 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6pcxm" event={"ID":"ae856da3-bc5b-43df-a4ec-45cae56db52f","Type":"ContainerDied","Data":"096627295a641646fbd295b19e87ccf9f4a0c71d64af3d14a8641d06bf5df19a"} Oct 30 23:11:52.515168 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:52.515147 4952 generic.go:356] "Generic (PLEG): container finished" podID="91651ce2-4673-4ced-b26d-e94ff8829d11" containerID="50e7fbffac004ee711615f94644a8755aca7d3d13b8274bb008cf80f7471f945" exitCode=0 Oct 30 23:11:52.515534 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:52.515512 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft" event={"ID":"91651ce2-4673-4ced-b26d-e94ff8829d11","Type":"ContainerDied","Data":"50e7fbffac004ee711615f94644a8755aca7d3d13b8274bb008cf80f7471f945"} Oct 30 23:11:52.517905 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:52.517887 4952 generic.go:356] "Generic (PLEG): container finished" podID="8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3" containerID="5c8fc771882cdfb9874bc3224ba5ec423bfb74f3b4e962dfdd37419e3af1f0fe" exitCode=0 Oct 30 23:11:52.517989 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:52.517947 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac2glm" event={"ID":"8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3","Type":"ContainerDied","Data":"5c8fc771882cdfb9874bc3224ba5ec423bfb74f3b4e962dfdd37419e3af1f0fe"} Oct 30 23:11:53.542336 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:53.540692 4952 generic.go:356] "Generic (PLEG): container finished" podID="4291c41e-29e3-49f1-9552-4a417d0bc0d2" containerID="cf26ee2cf67d123f45a3e44b5cefe69825bfa3f6ada5dea275900209e95d766e" exitCode=0 Oct 30 23:11:53.542336 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:53.542081 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nb95d" event={"ID":"4291c41e-29e3-49f1-9552-4a417d0bc0d2","Type":"ContainerDied","Data":"cf26ee2cf67d123f45a3e44b5cefe69825bfa3f6ada5dea275900209e95d766e"} Oct 30 23:11:57.612294 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:57.611227 4952 generic.go:356] "Generic (PLEG): container finished" podID="fbf82ede-9923-4c1e-a73a-9974c5eca3bc" containerID="cd69ea7c981373126e844f58dbc5fd6c36218e30a4dbd24558a244e92134c6d2" exitCode=0 Oct 30 23:11:57.612294 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:11:57.611296 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vlm92" event={"ID":"fbf82ede-9923-4c1e-a73a-9974c5eca3bc","Type":"ContainerDied","Data":"cd69ea7c981373126e844f58dbc5fd6c36218e30a4dbd24558a244e92134c6d2"} Oct 30 23:12:01.686220 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:01.686182 4952 generic.go:356] "Generic (PLEG): container finished" podID="a69b88b7-db93-4c88-9763-610ecf0e5590" containerID="88df49c52ed274d9a56738a7f37cfff0ed851990607b861b7561f33cedcdff80" exitCode=0 Oct 30 23:12:01.686725 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:01.686325 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd7cww" event={"ID":"a69b88b7-db93-4c88-9763-610ecf0e5590","Type":"ContainerDied","Data":"88df49c52ed274d9a56738a7f37cfff0ed851990607b861b7561f33cedcdff80"} Oct 30 23:12:12.402055 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:12.401926 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:15.906310 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:15.906031 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar4k5p" event={"ID":"e883e62d-ee03-411c-8fac-e9dacb0dc5e9","Type":"ContainerStarted","Data":"afabe32c8e732c1d74da8c18797755db1e4ae19ed2118d1fa67e685e68449806"} Oct 30 23:12:15.911924 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:15.911891 4952 generic.go:356] "Generic (PLEG): container finished" podID="b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41" containerID="ad6f79f10920477113c24ba2b1fcaf5e1a071239b3641b868109344bad5e3ec0" exitCode=0 Oct 30 23:12:15.912095 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:15.911997 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78trw2" event={"ID":"b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41","Type":"ContainerDied","Data":"ad6f79f10920477113c24ba2b1fcaf5e1a071239b3641b868109344bad5e3ec0"} Oct 30 23:12:15.915953 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:15.915480 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6pcxm" event={"ID":"ae856da3-bc5b-43df-a4ec-45cae56db52f","Type":"ContainerStarted","Data":"2a8b8fe2cf606a5b2d6e73735e0811feacedad681a84e4a43aa4ec7816423c90"} Oct 30 23:12:16.068057 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:16.067931 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar4k5p" podStartSLOduration=2.428090143 podStartE2EDuration="41.067912307s" totalImagesPullingTime="38.639822164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:11:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:12:16.067483137 +0000 UTC m=+355.211313342" watchObservedRunningTime="2026-10-30 23:12:16.067912307 +0000 UTC m=+355.211742504" Oct 30 23:12:16.184738 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:16.184693 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6pcxm" podStartSLOduration=2.536703802 podStartE2EDuration="48.184675819s" totalImagesPullingTime="45.647972017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:11:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:12:16.183149868 +0000 UTC m=+355.326980095" watchObservedRunningTime="2026-10-30 23:12:16.184675819 +0000 UTC m=+355.328506015" Oct 30 23:12:16.924496 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:16.924433 4952 generic.go:356] "Generic (PLEG): container finished" podID="a69b88b7-db93-4c88-9763-610ecf0e5590" containerID="bd9a3b5a411c698056616a25a4ca6af064a780aa494403e686d553a734cf5eb5" exitCode=0 Oct 30 23:12:16.924898 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:16.924523 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd7cww" event={"ID":"a69b88b7-db93-4c88-9763-610ecf0e5590","Type":"ContainerDied","Data":"bd9a3b5a411c698056616a25a4ca6af064a780aa494403e686d553a734cf5eb5"} Oct 30 23:12:16.926935 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:16.926909 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nb95d" event={"ID":"4291c41e-29e3-49f1-9552-4a417d0bc0d2","Type":"ContainerStarted","Data":"15ac873ea0072f7222040afb25999c84711fa3df98ed1440057126a9e27bc3e1"} Oct 30 23:12:16.929458 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:16.929438 4952 generic.go:356] "Generic (PLEG): container finished" podID="e883e62d-ee03-411c-8fac-e9dacb0dc5e9" containerID="afabe32c8e732c1d74da8c18797755db1e4ae19ed2118d1fa67e685e68449806" exitCode=0 Oct 30 23:12:16.929574 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:16.929467 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar4k5p" event={"ID":"e883e62d-ee03-411c-8fac-e9dacb0dc5e9","Type":"ContainerDied","Data":"afabe32c8e732c1d74da8c18797755db1e4ae19ed2118d1fa67e685e68449806"} Oct 30 23:12:16.933189 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:16.933166 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz7h28" event={"ID":"841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1","Type":"ContainerStarted","Data":"b1d20d39ecb819a0f6a9daff6291cc463aee1dd536ff8c6a7c61e4a65533c4a6"} Oct 30 23:12:16.935945 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:16.935844 4952 generic.go:356] "Generic (PLEG): container finished" podID="91fede71-5a6e-40fa-873b-16c20fedb00c" containerID="99b8c62c10040bbce0167354f4f7df2a04a169f204f767b4e995d9fa43d2e047" exitCode=0 Oct 30 23:12:16.935945 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:16.935925 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lbh9v" event={"ID":"91fede71-5a6e-40fa-873b-16c20fedb00c","Type":"ContainerDied","Data":"99b8c62c10040bbce0167354f4f7df2a04a169f204f767b4e995d9fa43d2e047"} Oct 30 23:12:16.938365 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:16.938346 4952 generic.go:356] "Generic (PLEG): container finished" podID="1166c90f-c4d5-4e7d-abd7-bde3f2c44010" containerID="d4a40418a8fd03351766bb9fd22b30ca6361f6aafabd124cad45e5be9af89f1a" exitCode=0 Oct 30 23:12:16.938456 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:16.938406 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tg9xk" event={"ID":"1166c90f-c4d5-4e7d-abd7-bde3f2c44010","Type":"ContainerDied","Data":"d4a40418a8fd03351766bb9fd22b30ca6361f6aafabd124cad45e5be9af89f1a"} Oct 30 23:12:16.940741 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:16.940715 4952 generic.go:356] "Generic (PLEG): container finished" podID="ae856da3-bc5b-43df-a4ec-45cae56db52f" containerID="2a8b8fe2cf606a5b2d6e73735e0811feacedad681a84e4a43aa4ec7816423c90" exitCode=0 Oct 30 23:12:16.940856 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:16.940750 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6pcxm" event={"ID":"ae856da3-bc5b-43df-a4ec-45cae56db52f","Type":"ContainerDied","Data":"2a8b8fe2cf606a5b2d6e73735e0811feacedad681a84e4a43aa4ec7816423c90"} Oct 30 23:12:16.942390 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:16.942374 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft_91651ce2-4673-4ced-b26d-e94ff8829d11/extract/0.log" Oct 30 23:12:16.943051 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:16.943011 4952 generic.go:356] "Generic (PLEG): container finished" podID="91651ce2-4673-4ced-b26d-e94ff8829d11" containerID="b9a09d68f8e7aa6fe26025bc1ddb4c501a4d047b5dcd586ba7c496bb32dec19e" exitCode=1 Oct 30 23:12:16.943147 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:16.943077 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft" event={"ID":"91651ce2-4673-4ced-b26d-e94ff8829d11","Type":"ContainerDied","Data":"b9a09d68f8e7aa6fe26025bc1ddb4c501a4d047b5dcd586ba7c496bb32dec19e"} Oct 30 23:12:16.945709 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:16.945587 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac2glm" event={"ID":"8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3","Type":"ContainerStarted","Data":"387fe2bc38631804900697591aa8e629ada7b9a50d805dceefd79f6a36fa2a91"} Oct 30 23:12:17.081982 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:17.081958 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78trw2" Oct 30 23:12:17.163287 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:17.163241 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz7h28" podStartSLOduration=3.446612163 podStartE2EDuration="42.163224227s" totalImagesPullingTime="38.716612064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:11:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:12:17.162367163 +0000 UTC m=+356.306197396" watchObservedRunningTime="2026-10-30 23:12:17.163224227 +0000 UTC m=+356.307054431" Oct 30 23:12:17.172582 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:17.172553 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41-bundle\") pod \"b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41\" (UID: \"b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41\") " Oct 30 23:12:17.172730 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:17.172600 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41-kube-api-access-b8nf8\" (UniqueName: \"kubernetes.io/projected/b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41-kube-api-access-b8nf8\") pod \"b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41\" (UID: \"b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41\") " Oct 30 23:12:17.172788 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:17.172742 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41-util\" (UniqueName: \"kubernetes.io/empty-dir/b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41-util\") pod \"b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41\" (UID: \"b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41\") " Oct 30 23:12:17.173241 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:17.173207 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41-bundle" pod "b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41" (UID: "b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:12:17.175346 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:17.175289 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41-kube-api-access-b8nf8" pod "b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41" (UID: "b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41"). InnerVolumeSpecName "kube-api-access-b8nf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:12:17.178556 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:17.178532 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41-util" pod "b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41" (UID: "b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:12:17.273968 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:17.273931 4952 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41-util\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:12:17.273968 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:17.273964 4952 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41-bundle\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:12:17.273968 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:17.273974 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8nf8\" (UniqueName: \"kubernetes.io/projected/b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41-kube-api-access-b8nf8\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:12:17.278321 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:17.278273 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nb95d" podStartSLOduration=3.273923636 podStartE2EDuration="47.278254138s" totalImagesPullingTime="44.004330502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:11:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:12:17.277136445 +0000 UTC m=+356.420966670" watchObservedRunningTime="2026-10-30 23:12:17.278254138 +0000 UTC m=+356.422084343" Oct 30 23:12:17.625705 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:17.625652 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac2glm" podStartSLOduration=4.101078319 podStartE2EDuration="50.625633493s" totalImagesPullingTime="46.524555174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:11:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:12:17.624833056 +0000 UTC m=+356.768663282" watchObservedRunningTime="2026-10-30 23:12:17.625633493 +0000 UTC m=+356.769463700" Oct 30 23:12:17.952062 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:17.951953 4952 generic.go:356] "Generic (PLEG): container finished" podID="841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1" containerID="b1d20d39ecb819a0f6a9daff6291cc463aee1dd536ff8c6a7c61e4a65533c4a6" exitCode=0 Oct 30 23:12:17.952551 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:17.952061 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz7h28" event={"ID":"841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1","Type":"ContainerDied","Data":"b1d20d39ecb819a0f6a9daff6291cc463aee1dd536ff8c6a7c61e4a65533c4a6"} Oct 30 23:12:17.954377 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:17.954352 4952 generic.go:356] "Generic (PLEG): container finished" podID="8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3" containerID="387fe2bc38631804900697591aa8e629ada7b9a50d805dceefd79f6a36fa2a91" exitCode=0 Oct 30 23:12:17.954508 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:17.954414 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac2glm" event={"ID":"8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3","Type":"ContainerDied","Data":"387fe2bc38631804900697591aa8e629ada7b9a50d805dceefd79f6a36fa2a91"} Oct 30 23:12:17.956569 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:17.956549 4952 generic.go:356] "Generic (PLEG): container finished" podID="4291c41e-29e3-49f1-9552-4a417d0bc0d2" containerID="15ac873ea0072f7222040afb25999c84711fa3df98ed1440057126a9e27bc3e1" exitCode=0 Oct 30 23:12:17.956669 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:17.956632 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nb95d" event={"ID":"4291c41e-29e3-49f1-9552-4a417d0bc0d2","Type":"ContainerDied","Data":"15ac873ea0072f7222040afb25999c84711fa3df98ed1440057126a9e27bc3e1"} Oct 30 23:12:17.958495 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:17.958478 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78trw2" Oct 30 23:12:17.958579 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:17.958522 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78trw2" event={"ID":"b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41","Type":"ContainerDied","Data":"936580b7e52d1a72468afc09535948877e45973411cab58d14335835f0e42c18"} Oct 30 23:12:17.958579 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:17.958561 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="936580b7e52d1a72468afc09535948877e45973411cab58d14335835f0e42c18" Oct 30 23:12:18.131753 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.131654 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tg9xk" Oct 30 23:12:18.212549 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.212462 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft_91651ce2-4673-4ced-b26d-e94ff8829d11/extract/0.log" Oct 30 23:12:18.215865 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.215806 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft" Oct 30 23:12:18.224942 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.224913 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar4k5p" Oct 30 23:12:18.231273 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.231248 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6pcxm" Oct 30 23:12:18.233436 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.233416 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd7cww" Oct 30 23:12:18.237010 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.236992 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lbh9v" Oct 30 23:12:18.282299 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.282276 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e883e62d-ee03-411c-8fac-e9dacb0dc5e9-kube-api-access-lcvvm\" (UniqueName: \"kubernetes.io/projected/e883e62d-ee03-411c-8fac-e9dacb0dc5e9-kube-api-access-lcvvm\") pod \"e883e62d-ee03-411c-8fac-e9dacb0dc5e9\" (UID: \"e883e62d-ee03-411c-8fac-e9dacb0dc5e9\") " Oct 30 23:12:18.282424 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.282344 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91651ce2-4673-4ced-b26d-e94ff8829d11-util\" (UniqueName: \"kubernetes.io/empty-dir/91651ce2-4673-4ced-b26d-e94ff8829d11-util\") pod \"91651ce2-4673-4ced-b26d-e94ff8829d11\" (UID: \"91651ce2-4673-4ced-b26d-e94ff8829d11\") " Oct 30 23:12:18.282424 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.282389 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e883e62d-ee03-411c-8fac-e9dacb0dc5e9-util\" (UniqueName: \"kubernetes.io/empty-dir/e883e62d-ee03-411c-8fac-e9dacb0dc5e9-util\") pod \"e883e62d-ee03-411c-8fac-e9dacb0dc5e9\" (UID: \"e883e62d-ee03-411c-8fac-e9dacb0dc5e9\") " Oct 30 23:12:18.282497 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.282430 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1166c90f-c4d5-4e7d-abd7-bde3f2c44010-kube-api-access-xsqzq\" (UniqueName: \"kubernetes.io/projected/1166c90f-c4d5-4e7d-abd7-bde3f2c44010-kube-api-access-xsqzq\") pod \"1166c90f-c4d5-4e7d-abd7-bde3f2c44010\" (UID: \"1166c90f-c4d5-4e7d-abd7-bde3f2c44010\") " Oct 30 23:12:18.282497 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.282464 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91651ce2-4673-4ced-b26d-e94ff8829d11-kube-api-access-7656f\" (UniqueName: \"kubernetes.io/projected/91651ce2-4673-4ced-b26d-e94ff8829d11-kube-api-access-7656f\") pod \"91651ce2-4673-4ced-b26d-e94ff8829d11\" (UID: \"91651ce2-4673-4ced-b26d-e94ff8829d11\") " Oct 30 23:12:18.282576 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.282496 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1166c90f-c4d5-4e7d-abd7-bde3f2c44010-util\" (UniqueName: \"kubernetes.io/empty-dir/1166c90f-c4d5-4e7d-abd7-bde3f2c44010-util\") pod \"1166c90f-c4d5-4e7d-abd7-bde3f2c44010\" (UID: \"1166c90f-c4d5-4e7d-abd7-bde3f2c44010\") " Oct 30 23:12:18.282576 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.282552 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1166c90f-c4d5-4e7d-abd7-bde3f2c44010-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1166c90f-c4d5-4e7d-abd7-bde3f2c44010-bundle\") pod \"1166c90f-c4d5-4e7d-abd7-bde3f2c44010\" (UID: \"1166c90f-c4d5-4e7d-abd7-bde3f2c44010\") " Oct 30 23:12:18.282679 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.282602 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e883e62d-ee03-411c-8fac-e9dacb0dc5e9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e883e62d-ee03-411c-8fac-e9dacb0dc5e9-bundle\") pod \"e883e62d-ee03-411c-8fac-e9dacb0dc5e9\" (UID: \"e883e62d-ee03-411c-8fac-e9dacb0dc5e9\") " Oct 30 23:12:18.282679 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.282641 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91651ce2-4673-4ced-b26d-e94ff8829d11-bundle\" (UniqueName: \"kubernetes.io/empty-dir/91651ce2-4673-4ced-b26d-e94ff8829d11-bundle\") pod \"91651ce2-4673-4ced-b26d-e94ff8829d11\" (UID: \"91651ce2-4673-4ced-b26d-e94ff8829d11\") " Oct 30 23:12:18.284194 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.283392 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1166c90f-c4d5-4e7d-abd7-bde3f2c44010-bundle" pod "1166c90f-c4d5-4e7d-abd7-bde3f2c44010" (UID: "1166c90f-c4d5-4e7d-abd7-bde3f2c44010"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:12:18.284194 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.283407 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91651ce2-4673-4ced-b26d-e94ff8829d11-bundle" pod "91651ce2-4673-4ced-b26d-e94ff8829d11" (UID: "91651ce2-4673-4ced-b26d-e94ff8829d11"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:12:18.284194 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.284165 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e883e62d-ee03-411c-8fac-e9dacb0dc5e9-bundle" pod "e883e62d-ee03-411c-8fac-e9dacb0dc5e9" (UID: "e883e62d-ee03-411c-8fac-e9dacb0dc5e9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:12:18.284964 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.284935 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e883e62d-ee03-411c-8fac-e9dacb0dc5e9-kube-api-access-lcvvm" pod "e883e62d-ee03-411c-8fac-e9dacb0dc5e9" (UID: "e883e62d-ee03-411c-8fac-e9dacb0dc5e9"). InnerVolumeSpecName "kube-api-access-lcvvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:12:18.285199 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.285174 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1166c90f-c4d5-4e7d-abd7-bde3f2c44010-kube-api-access-xsqzq" pod "1166c90f-c4d5-4e7d-abd7-bde3f2c44010" (UID: "1166c90f-c4d5-4e7d-abd7-bde3f2c44010"). InnerVolumeSpecName "kube-api-access-xsqzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:12:18.285287 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.285241 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91651ce2-4673-4ced-b26d-e94ff8829d11-kube-api-access-7656f" pod "91651ce2-4673-4ced-b26d-e94ff8829d11" (UID: "91651ce2-4673-4ced-b26d-e94ff8829d11"). InnerVolumeSpecName "kube-api-access-7656f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:12:18.288355 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.288333 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e883e62d-ee03-411c-8fac-e9dacb0dc5e9-util" pod "e883e62d-ee03-411c-8fac-e9dacb0dc5e9" (UID: "e883e62d-ee03-411c-8fac-e9dacb0dc5e9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:12:18.288745 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.288725 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1166c90f-c4d5-4e7d-abd7-bde3f2c44010-util" pod "1166c90f-c4d5-4e7d-abd7-bde3f2c44010" (UID: "1166c90f-c4d5-4e7d-abd7-bde3f2c44010"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:12:18.289181 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.289161 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91651ce2-4673-4ced-b26d-e94ff8829d11-util" pod "91651ce2-4673-4ced-b26d-e94ff8829d11" (UID: "91651ce2-4673-4ced-b26d-e94ff8829d11"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:12:18.383462 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.383439 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a69b88b7-db93-4c88-9763-610ecf0e5590-util\" (UniqueName: \"kubernetes.io/empty-dir/a69b88b7-db93-4c88-9763-610ecf0e5590-util\") pod \"a69b88b7-db93-4c88-9763-610ecf0e5590\" (UID: \"a69b88b7-db93-4c88-9763-610ecf0e5590\") " Oct 30 23:12:18.383571 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.383489 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae856da3-bc5b-43df-a4ec-45cae56db52f-kube-api-access-64mrb\" (UniqueName: \"kubernetes.io/projected/ae856da3-bc5b-43df-a4ec-45cae56db52f-kube-api-access-64mrb\") pod \"ae856da3-bc5b-43df-a4ec-45cae56db52f\" (UID: \"ae856da3-bc5b-43df-a4ec-45cae56db52f\") " Oct 30 23:12:18.383571 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.383513 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae856da3-bc5b-43df-a4ec-45cae56db52f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae856da3-bc5b-43df-a4ec-45cae56db52f-bundle\") pod \"ae856da3-bc5b-43df-a4ec-45cae56db52f\" (UID: \"ae856da3-bc5b-43df-a4ec-45cae56db52f\") " Oct 30 23:12:18.383571 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.383556 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91fede71-5a6e-40fa-873b-16c20fedb00c-util\" (UniqueName: \"kubernetes.io/empty-dir/91fede71-5a6e-40fa-873b-16c20fedb00c-util\") pod \"91fede71-5a6e-40fa-873b-16c20fedb00c\" (UID: \"91fede71-5a6e-40fa-873b-16c20fedb00c\") " Oct 30 23:12:18.383683 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.383586 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae856da3-bc5b-43df-a4ec-45cae56db52f-util\" (UniqueName: \"kubernetes.io/empty-dir/ae856da3-bc5b-43df-a4ec-45cae56db52f-util\") pod \"ae856da3-bc5b-43df-a4ec-45cae56db52f\" (UID: \"ae856da3-bc5b-43df-a4ec-45cae56db52f\") " Oct 30 23:12:18.383683 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.383652 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91fede71-5a6e-40fa-873b-16c20fedb00c-kube-api-access-l6222\" (UniqueName: \"kubernetes.io/projected/91fede71-5a6e-40fa-873b-16c20fedb00c-kube-api-access-l6222\") pod \"91fede71-5a6e-40fa-873b-16c20fedb00c\" (UID: \"91fede71-5a6e-40fa-873b-16c20fedb00c\") " Oct 30 23:12:18.383785 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.383712 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a69b88b7-db93-4c88-9763-610ecf0e5590-kube-api-access-d7s5m\" (UniqueName: \"kubernetes.io/projected/a69b88b7-db93-4c88-9763-610ecf0e5590-kube-api-access-d7s5m\") pod \"a69b88b7-db93-4c88-9763-610ecf0e5590\" (UID: \"a69b88b7-db93-4c88-9763-610ecf0e5590\") " Oct 30 23:12:18.383785 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.383746 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91fede71-5a6e-40fa-873b-16c20fedb00c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/91fede71-5a6e-40fa-873b-16c20fedb00c-bundle\") pod \"91fede71-5a6e-40fa-873b-16c20fedb00c\" (UID: \"91fede71-5a6e-40fa-873b-16c20fedb00c\") " Oct 30 23:12:18.383894 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.383790 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a69b88b7-db93-4c88-9763-610ecf0e5590-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a69b88b7-db93-4c88-9763-610ecf0e5590-bundle\") pod \"a69b88b7-db93-4c88-9763-610ecf0e5590\" (UID: \"a69b88b7-db93-4c88-9763-610ecf0e5590\") " Oct 30 23:12:18.384270 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.384090 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsqzq\" (UniqueName: \"kubernetes.io/projected/1166c90f-c4d5-4e7d-abd7-bde3f2c44010-kube-api-access-xsqzq\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:12:18.384270 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.384116 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7656f\" (UniqueName: \"kubernetes.io/projected/91651ce2-4673-4ced-b26d-e94ff8829d11-kube-api-access-7656f\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:12:18.384270 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.384132 4952 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1166c90f-c4d5-4e7d-abd7-bde3f2c44010-util\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:12:18.384270 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.384145 4952 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1166c90f-c4d5-4e7d-abd7-bde3f2c44010-bundle\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:12:18.384270 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.384144 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae856da3-bc5b-43df-a4ec-45cae56db52f-bundle" pod "ae856da3-bc5b-43df-a4ec-45cae56db52f" (UID: "ae856da3-bc5b-43df-a4ec-45cae56db52f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:12:18.384270 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.384158 4952 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e883e62d-ee03-411c-8fac-e9dacb0dc5e9-bundle\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:12:18.384270 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.384196 4952 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/91651ce2-4673-4ced-b26d-e94ff8829d11-bundle\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:12:18.384270 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.384213 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcvvm\" (UniqueName: \"kubernetes.io/projected/e883e62d-ee03-411c-8fac-e9dacb0dc5e9-kube-api-access-lcvvm\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:12:18.384681 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.384529 4952 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/91651ce2-4673-4ced-b26d-e94ff8829d11-util\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:12:18.384681 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.384546 4952 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e883e62d-ee03-411c-8fac-e9dacb0dc5e9-util\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:12:18.384910 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.384864 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fede71-5a6e-40fa-873b-16c20fedb00c-bundle" pod "91fede71-5a6e-40fa-873b-16c20fedb00c" (UID: "91fede71-5a6e-40fa-873b-16c20fedb00c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:12:18.385047 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.385013 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a69b88b7-db93-4c88-9763-610ecf0e5590-bundle" pod "a69b88b7-db93-4c88-9763-610ecf0e5590" (UID: "a69b88b7-db93-4c88-9763-610ecf0e5590"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:12:18.386071 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.386010 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae856da3-bc5b-43df-a4ec-45cae56db52f-kube-api-access-64mrb" pod "ae856da3-bc5b-43df-a4ec-45cae56db52f" (UID: "ae856da3-bc5b-43df-a4ec-45cae56db52f"). InnerVolumeSpecName "kube-api-access-64mrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:12:18.386485 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.386462 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91fede71-5a6e-40fa-873b-16c20fedb00c-kube-api-access-l6222" pod "91fede71-5a6e-40fa-873b-16c20fedb00c" (UID: "91fede71-5a6e-40fa-873b-16c20fedb00c"). InnerVolumeSpecName "kube-api-access-l6222". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:12:18.386858 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.386835 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a69b88b7-db93-4c88-9763-610ecf0e5590-kube-api-access-d7s5m" pod "a69b88b7-db93-4c88-9763-610ecf0e5590" (UID: "a69b88b7-db93-4c88-9763-610ecf0e5590"). InnerVolumeSpecName "kube-api-access-d7s5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:12:18.391580 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.391555 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae856da3-bc5b-43df-a4ec-45cae56db52f-util" pod "ae856da3-bc5b-43df-a4ec-45cae56db52f" (UID: "ae856da3-bc5b-43df-a4ec-45cae56db52f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:12:18.391693 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.391579 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a69b88b7-db93-4c88-9763-610ecf0e5590-util" pod "a69b88b7-db93-4c88-9763-610ecf0e5590" (UID: "a69b88b7-db93-4c88-9763-610ecf0e5590"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:12:18.391944 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.391925 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fede71-5a6e-40fa-873b-16c20fedb00c-util" pod "91fede71-5a6e-40fa-873b-16c20fedb00c" (UID: "91fede71-5a6e-40fa-873b-16c20fedb00c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:12:18.486010 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.485950 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64mrb\" (UniqueName: \"kubernetes.io/projected/ae856da3-bc5b-43df-a4ec-45cae56db52f-kube-api-access-64mrb\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:12:18.486010 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.485974 4952 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae856da3-bc5b-43df-a4ec-45cae56db52f-bundle\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:12:18.486010 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.485983 4952 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/91fede71-5a6e-40fa-873b-16c20fedb00c-util\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:12:18.486010 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.485991 4952 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae856da3-bc5b-43df-a4ec-45cae56db52f-util\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:12:18.486010 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.485999 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6222\" (UniqueName: \"kubernetes.io/projected/91fede71-5a6e-40fa-873b-16c20fedb00c-kube-api-access-l6222\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:12:18.486010 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.486009 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7s5m\" (UniqueName: \"kubernetes.io/projected/a69b88b7-db93-4c88-9763-610ecf0e5590-kube-api-access-d7s5m\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:12:18.486010 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.486017 4952 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/91fede71-5a6e-40fa-873b-16c20fedb00c-bundle\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:12:18.486315 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.486026 4952 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a69b88b7-db93-4c88-9763-610ecf0e5590-bundle\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:12:18.486315 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.486046 4952 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a69b88b7-db93-4c88-9763-610ecf0e5590-util\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:12:18.964831 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.964805 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd7cww" Oct 30 23:12:18.967349 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.967319 4952 generic.go:356] "Generic (PLEG): container finished" podID="fbf82ede-9923-4c1e-a73a-9974c5eca3bc" containerID="193375ec44e53d34e389c7e0b0968d544dd974b820e3178be75802717278e6ba" exitCode=0 Oct 30 23:12:18.968953 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.968920 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cd7cww" event={"ID":"a69b88b7-db93-4c88-9763-610ecf0e5590","Type":"ContainerDied","Data":"d439968e3d8120e4324f5a7b7ffdf425288670c1ef06248e901b7b684c0a9ad3"} Oct 30 23:12:18.969113 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.968970 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d439968e3d8120e4324f5a7b7ffdf425288670c1ef06248e901b7b684c0a9ad3" Oct 30 23:12:18.969113 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.968988 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vlm92" event={"ID":"fbf82ede-9923-4c1e-a73a-9974c5eca3bc","Type":"ContainerDied","Data":"193375ec44e53d34e389c7e0b0968d544dd974b820e3178be75802717278e6ba"} Oct 30 23:12:18.976878 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.976819 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar4k5p" event={"ID":"e883e62d-ee03-411c-8fac-e9dacb0dc5e9","Type":"ContainerDied","Data":"e9e88b723e964362c2539832911ef31e441bf0cfd51a813d9efe0c33fe1dd27b"} Oct 30 23:12:18.976935 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.976892 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9e88b723e964362c2539832911ef31e441bf0cfd51a813d9efe0c33fe1dd27b" Oct 30 23:12:18.977056 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.977017 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ar4k5p" Oct 30 23:12:18.984444 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.984419 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lbh9v" event={"ID":"91fede71-5a6e-40fa-873b-16c20fedb00c","Type":"ContainerDied","Data":"07dde687075c97ffcb4a062ad5b51e7adb0780358b488447c46f4281305b910b"} Oct 30 23:12:18.984865 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.984848 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07dde687075c97ffcb4a062ad5b51e7adb0780358b488447c46f4281305b910b" Oct 30 23:12:18.985268 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.985195 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lbh9v" Oct 30 23:12:18.987573 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.987550 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tg9xk" event={"ID":"1166c90f-c4d5-4e7d-abd7-bde3f2c44010","Type":"ContainerDied","Data":"a7c896ce476d4f962911a68f82a869dea8e26d1b6a1b707b1cd732caff939e48"} Oct 30 23:12:18.987674 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.987573 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tg9xk" Oct 30 23:12:18.987674 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.987592 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7c896ce476d4f962911a68f82a869dea8e26d1b6a1b707b1cd732caff939e48" Oct 30 23:12:18.990247 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.990226 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6pcxm" event={"ID":"ae856da3-bc5b-43df-a4ec-45cae56db52f","Type":"ContainerDied","Data":"b68365b526fb488d72bd4ad3701910228e1d7963501f67362dc0af36b92c60f6"} Oct 30 23:12:18.990403 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.990381 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b68365b526fb488d72bd4ad3701910228e1d7963501f67362dc0af36b92c60f6" Oct 30 23:12:18.990512 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.990241 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6pcxm" Oct 30 23:12:18.991674 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.991658 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft_91651ce2-4673-4ced-b26d-e94ff8829d11/extract/0.log" Oct 30 23:12:18.992555 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.992537 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft" Oct 30 23:12:18.992655 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.992569 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft" event={"ID":"91651ce2-4673-4ced-b26d-e94ff8829d11","Type":"ContainerDied","Data":"c9602329f18665a19be7aafa042d3cd75b799e74793c54bc7eca9bd7efcd72f1"} Oct 30 23:12:18.992655 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:18.992601 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9602329f18665a19be7aafa042d3cd75b799e74793c54bc7eca9bd7efcd72f1" Oct 30 23:12:19.136411 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:19.136387 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz7h28" Oct 30 23:12:19.163506 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:19.163480 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac2glm" Oct 30 23:12:19.197392 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:19.197370 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nb95d" Oct 30 23:12:19.292495 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:19.292432 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1-kube-api-access-tcxd4\" (UniqueName: \"kubernetes.io/projected/841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1-kube-api-access-tcxd4\") pod \"841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1\" (UID: \"841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1\") " Oct 30 23:12:19.292495 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:19.292480 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3-bundle\") pod \"8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3\" (UID: \"8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3\") " Oct 30 23:12:19.292657 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:19.292512 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4291c41e-29e3-49f1-9552-4a417d0bc0d2-kube-api-access-s472n\" (UniqueName: \"kubernetes.io/projected/4291c41e-29e3-49f1-9552-4a417d0bc0d2-kube-api-access-s472n\") pod \"4291c41e-29e3-49f1-9552-4a417d0bc0d2\" (UID: \"4291c41e-29e3-49f1-9552-4a417d0bc0d2\") " Oct 30 23:12:19.292657 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:19.292633 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4291c41e-29e3-49f1-9552-4a417d0bc0d2-util\" (UniqueName: \"kubernetes.io/empty-dir/4291c41e-29e3-49f1-9552-4a417d0bc0d2-util\") pod \"4291c41e-29e3-49f1-9552-4a417d0bc0d2\" (UID: \"4291c41e-29e3-49f1-9552-4a417d0bc0d2\") " Oct 30 23:12:19.292766 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:19.292684 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1-bundle\") pod \"841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1\" (UID: \"841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1\") " Oct 30 23:12:19.292766 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:19.292717 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3-kube-api-access-2xtlg\" (UniqueName: \"kubernetes.io/projected/8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3-kube-api-access-2xtlg\") pod \"8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3\" (UID: \"8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3\") " Oct 30 23:12:19.292766 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:19.292762 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3-util\" (UniqueName: \"kubernetes.io/empty-dir/8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3-util\") pod \"8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3\" (UID: \"8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3\") " Oct 30 23:12:19.292908 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:19.292832 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4291c41e-29e3-49f1-9552-4a417d0bc0d2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4291c41e-29e3-49f1-9552-4a417d0bc0d2-bundle\") pod \"4291c41e-29e3-49f1-9552-4a417d0bc0d2\" (UID: \"4291c41e-29e3-49f1-9552-4a417d0bc0d2\") " Oct 30 23:12:19.292908 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:19.292857 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1-util\" (UniqueName: \"kubernetes.io/empty-dir/841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1-util\") pod \"841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1\" (UID: \"841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1\") " Oct 30 23:12:19.293733 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:19.293700 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4291c41e-29e3-49f1-9552-4a417d0bc0d2-bundle" pod "4291c41e-29e3-49f1-9552-4a417d0bc0d2" (UID: "4291c41e-29e3-49f1-9552-4a417d0bc0d2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:12:19.293733 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:19.293718 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3-bundle" pod "8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3" (UID: "8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:12:19.293916 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:19.293712 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1-bundle" pod "841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1" (UID: "841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:12:19.294982 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:19.294958 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4291c41e-29e3-49f1-9552-4a417d0bc0d2-kube-api-access-s472n" pod "4291c41e-29e3-49f1-9552-4a417d0bc0d2" (UID: "4291c41e-29e3-49f1-9552-4a417d0bc0d2"). InnerVolumeSpecName "kube-api-access-s472n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:12:19.295183 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:19.295166 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1-kube-api-access-tcxd4" pod "841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1" (UID: "841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1"). InnerVolumeSpecName "kube-api-access-tcxd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:12:19.296099 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:19.296080 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3-kube-api-access-2xtlg" pod "8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3" (UID: "8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3"). InnerVolumeSpecName "kube-api-access-2xtlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:12:19.299562 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:19.299544 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4291c41e-29e3-49f1-9552-4a417d0bc0d2-util" pod "4291c41e-29e3-49f1-9552-4a417d0bc0d2" (UID: "4291c41e-29e3-49f1-9552-4a417d0bc0d2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:12:19.301816 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:19.301794 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1-util" pod "841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1" (UID: "841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:12:19.302423 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:19.302407 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3-util" pod "8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3" (UID: "8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:12:19.393937 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:19.393917 4952 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3-bundle\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:12:19.394083 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:19.393943 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s472n\" (UniqueName: \"kubernetes.io/projected/4291c41e-29e3-49f1-9552-4a417d0bc0d2-kube-api-access-s472n\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:12:19.394083 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:19.393957 4952 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4291c41e-29e3-49f1-9552-4a417d0bc0d2-util\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:12:19.394083 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:19.393969 4952 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1-bundle\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:12:19.394083 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:19.393982 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xtlg\" (UniqueName: \"kubernetes.io/projected/8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3-kube-api-access-2xtlg\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:12:19.394083 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:19.393995 4952 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3-util\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:12:19.394083 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:19.394008 4952 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4291c41e-29e3-49f1-9552-4a417d0bc0d2-bundle\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:12:19.394083 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:19.394019 4952 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1-util\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:12:19.394083 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:19.394031 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcxd4\" (UniqueName: \"kubernetes.io/projected/841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1-kube-api-access-tcxd4\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:12:19.999559 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:19.999531 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac2glm" Oct 30 23:12:19.999559 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:19.999546 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac2glm" event={"ID":"8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3","Type":"ContainerDied","Data":"2bbbd3c386aa87335d8f04b7420be3ce26b77b13793e42f4b5057e405cf4c657"} Oct 30 23:12:20.000161 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:19.999591 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bbbd3c386aa87335d8f04b7420be3ce26b77b13793e42f4b5057e405cf4c657" Oct 30 23:12:20.001521 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:20.001499 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nb95d" Oct 30 23:12:20.001521 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:20.001510 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nb95d" event={"ID":"4291c41e-29e3-49f1-9552-4a417d0bc0d2","Type":"ContainerDied","Data":"a60e98aac11cac0274decd73e940fc63752e4568326c84e42723b60b35c792d3"} Oct 30 23:12:20.001709 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:20.001549 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a60e98aac11cac0274decd73e940fc63752e4568326c84e42723b60b35c792d3" Oct 30 23:12:20.003735 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:20.003706 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz7h28" Oct 30 23:12:20.003855 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:20.003737 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz7h28" event={"ID":"841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1","Type":"ContainerDied","Data":"fb23622cc446e2aef00b392e0171ee9edd80c5cd4c4edcd1c17d8076307cd5c4"} Oct 30 23:12:20.003855 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:20.003783 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb23622cc446e2aef00b392e0171ee9edd80c5cd4c4edcd1c17d8076307cd5c4" Oct 30 23:12:20.133167 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:20.133142 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vlm92" Oct 30 23:12:20.199591 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:20.199564 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbf82ede-9923-4c1e-a73a-9974c5eca3bc-kube-api-access-zvs64\" (UniqueName: \"kubernetes.io/projected/fbf82ede-9923-4c1e-a73a-9974c5eca3bc-kube-api-access-zvs64\") pod \"fbf82ede-9923-4c1e-a73a-9974c5eca3bc\" (UID: \"fbf82ede-9923-4c1e-a73a-9974c5eca3bc\") " Oct 30 23:12:20.199705 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:20.199596 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbf82ede-9923-4c1e-a73a-9974c5eca3bc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbf82ede-9923-4c1e-a73a-9974c5eca3bc-bundle\") pod \"fbf82ede-9923-4c1e-a73a-9974c5eca3bc\" (UID: \"fbf82ede-9923-4c1e-a73a-9974c5eca3bc\") " Oct 30 23:12:20.199705 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:20.199625 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbf82ede-9923-4c1e-a73a-9974c5eca3bc-util\" (UniqueName: \"kubernetes.io/empty-dir/fbf82ede-9923-4c1e-a73a-9974c5eca3bc-util\") pod \"fbf82ede-9923-4c1e-a73a-9974c5eca3bc\" (UID: \"fbf82ede-9923-4c1e-a73a-9974c5eca3bc\") " Oct 30 23:12:20.200051 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:20.200018 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbf82ede-9923-4c1e-a73a-9974c5eca3bc-bundle" pod "fbf82ede-9923-4c1e-a73a-9974c5eca3bc" (UID: "fbf82ede-9923-4c1e-a73a-9974c5eca3bc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:12:20.201781 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:20.201761 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbf82ede-9923-4c1e-a73a-9974c5eca3bc-kube-api-access-zvs64" pod "fbf82ede-9923-4c1e-a73a-9974c5eca3bc" (UID: "fbf82ede-9923-4c1e-a73a-9974c5eca3bc"). InnerVolumeSpecName "kube-api-access-zvs64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:12:20.205247 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:20.205223 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbf82ede-9923-4c1e-a73a-9974c5eca3bc-util" pod "fbf82ede-9923-4c1e-a73a-9974c5eca3bc" (UID: "fbf82ede-9923-4c1e-a73a-9974c5eca3bc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:12:20.301216 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:20.301151 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvs64\" (UniqueName: \"kubernetes.io/projected/fbf82ede-9923-4c1e-a73a-9974c5eca3bc-kube-api-access-zvs64\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:12:20.301216 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:20.301173 4952 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbf82ede-9923-4c1e-a73a-9974c5eca3bc-bundle\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:12:20.301216 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:20.301183 4952 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbf82ede-9923-4c1e-a73a-9974c5eca3bc-util\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:12:21.011721 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:21.011692 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vlm92" Oct 30 23:12:21.011721 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:21.011701 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vlm92" event={"ID":"fbf82ede-9923-4c1e-a73a-9974c5eca3bc","Type":"ContainerDied","Data":"dc0d10bc8098f56e85f16d1a966edc926416585f1a1878a7a5fca8e8de57b388"} Oct 30 23:12:21.012122 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:21.011766 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc0d10bc8098f56e85f16d1a966edc926416585f1a1878a7a5fca8e8de57b388" Oct 30 23:12:21.140086 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:12:21.140029 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:12:23.423803 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:23.423768 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:23.659831 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:23.659798 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:24.310616 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.310539 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:24.463732 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.463697 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-97n94"] Oct 30 23:12:24.464134 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464013 4952 cpu_manager.go:414] "containerMap: removing container" podUID="8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3" containerName="util" Oct 30 23:12:24.464134 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464023 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3" containerName="util" Oct 30 23:12:24.464134 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464032 4952 cpu_manager.go:414] "containerMap: removing container" podUID="ae856da3-bc5b-43df-a4ec-45cae56db52f" containerName="pull" Oct 30 23:12:24.464134 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464055 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae856da3-bc5b-43df-a4ec-45cae56db52f" containerName="pull" Oct 30 23:12:24.464134 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464072 4952 cpu_manager.go:414] "containerMap: removing container" podUID="91651ce2-4673-4ced-b26d-e94ff8829d11" containerName="util" Oct 30 23:12:24.464134 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464083 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91651ce2-4673-4ced-b26d-e94ff8829d11" containerName="util" Oct 30 23:12:24.464134 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464091 4952 cpu_manager.go:414] "containerMap: removing container" podUID="a69b88b7-db93-4c88-9763-610ecf0e5590" containerName="util" Oct 30 23:12:24.464134 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464097 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a69b88b7-db93-4c88-9763-610ecf0e5590" containerName="util" Oct 30 23:12:24.464134 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464108 4952 cpu_manager.go:414] "containerMap: removing container" podUID="8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3" containerName="pull" Oct 30 23:12:24.464134 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464115 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3" containerName="pull" Oct 30 23:12:24.464134 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464128 4952 cpu_manager.go:414] "containerMap: removing container" podUID="a69b88b7-db93-4c88-9763-610ecf0e5590" containerName="extract" Oct 30 23:12:24.464134 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464133 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a69b88b7-db93-4c88-9763-610ecf0e5590" containerName="extract" Oct 30 23:12:24.464134 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464140 4952 cpu_manager.go:414] "containerMap: removing container" podUID="1166c90f-c4d5-4e7d-abd7-bde3f2c44010" containerName="extract" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464144 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1166c90f-c4d5-4e7d-abd7-bde3f2c44010" containerName="extract" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464150 4952 cpu_manager.go:414] "containerMap: removing container" podUID="fbf82ede-9923-4c1e-a73a-9974c5eca3bc" containerName="extract" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464154 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbf82ede-9923-4c1e-a73a-9974c5eca3bc" containerName="extract" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464161 4952 cpu_manager.go:414] "containerMap: removing container" podUID="841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1" containerName="pull" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464165 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1" containerName="pull" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464171 4952 cpu_manager.go:414] "containerMap: removing container" podUID="e883e62d-ee03-411c-8fac-e9dacb0dc5e9" containerName="pull" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464178 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e883e62d-ee03-411c-8fac-e9dacb0dc5e9" containerName="pull" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464187 4952 cpu_manager.go:414] "containerMap: removing container" podUID="b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41" containerName="extract" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464194 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41" containerName="extract" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464202 4952 cpu_manager.go:414] "containerMap: removing container" podUID="91651ce2-4673-4ced-b26d-e94ff8829d11" containerName="extract" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464208 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91651ce2-4673-4ced-b26d-e94ff8829d11" containerName="extract" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464221 4952 cpu_manager.go:414] "containerMap: removing container" podUID="e883e62d-ee03-411c-8fac-e9dacb0dc5e9" containerName="extract" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464232 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e883e62d-ee03-411c-8fac-e9dacb0dc5e9" containerName="extract" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464241 4952 cpu_manager.go:414] "containerMap: removing container" podUID="1166c90f-c4d5-4e7d-abd7-bde3f2c44010" containerName="pull" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464248 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1166c90f-c4d5-4e7d-abd7-bde3f2c44010" containerName="pull" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464258 4952 cpu_manager.go:414] "containerMap: removing container" podUID="841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1" containerName="extract" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464264 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1" containerName="extract" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464279 4952 cpu_manager.go:414] "containerMap: removing container" podUID="8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3" containerName="extract" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464286 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3" containerName="extract" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464300 4952 cpu_manager.go:414] "containerMap: removing container" podUID="91fede71-5a6e-40fa-873b-16c20fedb00c" containerName="util" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464306 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91fede71-5a6e-40fa-873b-16c20fedb00c" containerName="util" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464318 4952 cpu_manager.go:414] "containerMap: removing container" podUID="ae856da3-bc5b-43df-a4ec-45cae56db52f" containerName="extract" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464326 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae856da3-bc5b-43df-a4ec-45cae56db52f" containerName="extract" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464341 4952 cpu_manager.go:414] "containerMap: removing container" podUID="b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41" containerName="util" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464349 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41" containerName="util" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464357 4952 cpu_manager.go:414] "containerMap: removing container" podUID="b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41" containerName="pull" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464363 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41" containerName="pull" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464373 4952 cpu_manager.go:414] "containerMap: removing container" podUID="91fede71-5a6e-40fa-873b-16c20fedb00c" containerName="extract" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464380 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91fede71-5a6e-40fa-873b-16c20fedb00c" containerName="extract" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464395 4952 cpu_manager.go:414] "containerMap: removing container" podUID="91651ce2-4673-4ced-b26d-e94ff8829d11" containerName="pull" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464402 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91651ce2-4673-4ced-b26d-e94ff8829d11" containerName="pull" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464411 4952 cpu_manager.go:414] "containerMap: removing container" podUID="4291c41e-29e3-49f1-9552-4a417d0bc0d2" containerName="pull" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464417 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4291c41e-29e3-49f1-9552-4a417d0bc0d2" containerName="pull" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464425 4952 cpu_manager.go:414] "containerMap: removing container" podUID="fbf82ede-9923-4c1e-a73a-9974c5eca3bc" containerName="pull" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464431 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbf82ede-9923-4c1e-a73a-9974c5eca3bc" containerName="pull" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464446 4952 cpu_manager.go:414] "containerMap: removing container" podUID="a69b88b7-db93-4c88-9763-610ecf0e5590" containerName="pull" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464452 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a69b88b7-db93-4c88-9763-610ecf0e5590" containerName="pull" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464461 4952 cpu_manager.go:414] "containerMap: removing container" podUID="4291c41e-29e3-49f1-9552-4a417d0bc0d2" containerName="extract" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464468 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4291c41e-29e3-49f1-9552-4a417d0bc0d2" containerName="extract" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464479 4952 cpu_manager.go:414] "containerMap: removing container" podUID="1166c90f-c4d5-4e7d-abd7-bde3f2c44010" containerName="util" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464485 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1166c90f-c4d5-4e7d-abd7-bde3f2c44010" containerName="util" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464494 4952 cpu_manager.go:414] "containerMap: removing container" podUID="91fede71-5a6e-40fa-873b-16c20fedb00c" containerName="pull" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464501 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91fede71-5a6e-40fa-873b-16c20fedb00c" containerName="pull" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464516 4952 cpu_manager.go:414] "containerMap: removing container" podUID="e883e62d-ee03-411c-8fac-e9dacb0dc5e9" containerName="util" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464523 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e883e62d-ee03-411c-8fac-e9dacb0dc5e9" containerName="util" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464535 4952 cpu_manager.go:414] "containerMap: removing container" podUID="ae856da3-bc5b-43df-a4ec-45cae56db52f" containerName="util" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464542 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae856da3-bc5b-43df-a4ec-45cae56db52f" containerName="util" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464553 4952 cpu_manager.go:414] "containerMap: removing container" podUID="fbf82ede-9923-4c1e-a73a-9974c5eca3bc" containerName="util" Oct 30 23:12:24.464540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464560 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbf82ede-9923-4c1e-a73a-9974c5eca3bc" containerName="util" Oct 30 23:12:24.465834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464566 4952 cpu_manager.go:414] "containerMap: removing container" podUID="841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1" containerName="util" Oct 30 23:12:24.465834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464571 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1" containerName="util" Oct 30 23:12:24.465834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464576 4952 cpu_manager.go:414] "containerMap: removing container" podUID="4291c41e-29e3-49f1-9552-4a417d0bc0d2" containerName="util" Oct 30 23:12:24.465834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464580 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4291c41e-29e3-49f1-9552-4a417d0bc0d2" containerName="util" Oct 30 23:12:24.465834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464657 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="4291c41e-29e3-49f1-9552-4a417d0bc0d2" containerName="extract" Oct 30 23:12:24.465834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464668 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0bbb2ec-d17f-4b9f-8fa0-8affd41bbc41" containerName="extract" Oct 30 23:12:24.465834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464674 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="91651ce2-4673-4ced-b26d-e94ff8829d11" containerName="extract" Oct 30 23:12:24.465834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464680 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="a69b88b7-db93-4c88-9763-610ecf0e5590" containerName="extract" Oct 30 23:12:24.465834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464686 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="1166c90f-c4d5-4e7d-abd7-bde3f2c44010" containerName="extract" Oct 30 23:12:24.465834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464690 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbf82ede-9923-4c1e-a73a-9974c5eca3bc" containerName="extract" Oct 30 23:12:24.465834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464695 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="e883e62d-ee03-411c-8fac-e9dacb0dc5e9" containerName="extract" Oct 30 23:12:24.465834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464700 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="841bd0a8-32f9-4ea8-afe4-1a8127a5a1f1" containerName="extract" Oct 30 23:12:24.465834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464705 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e1aa4b4-753a-428e-bd78-fd9f2b7b8ab3" containerName="extract" Oct 30 23:12:24.465834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464710 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae856da3-bc5b-43df-a4ec-45cae56db52f" containerName="extract" Oct 30 23:12:24.465834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.464716 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="91fede71-5a6e-40fa-873b-16c20fedb00c" containerName="extract" Oct 30 23:12:24.467195 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.467179 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-97n94" Oct 30 23:12:24.532344 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.532317 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bc83e08c-8b67-47b0-834d-bbb30387a45b-manifests\") pod \"opendatahub-operator-controller-manager-5b8f94678b-97n94\" (UID: \"bc83e08c-8b67-47b0-834d-bbb30387a45b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-97n94" Oct 30 23:12:24.532469 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.532352 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bc83e08c-8b67-47b0-834d-bbb30387a45b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b8f94678b-97n94\" (UID: \"bc83e08c-8b67-47b0-834d-bbb30387a45b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-97n94" Oct 30 23:12:24.532510 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.532464 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bc83e08c-8b67-47b0-834d-bbb30387a45b-metrics-cert\") pod \"opendatahub-operator-controller-manager-5b8f94678b-97n94\" (UID: \"bc83e08c-8b67-47b0-834d-bbb30387a45b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-97n94" Oct 30 23:12:24.532510 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.532488 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pkqf\" (UniqueName: \"kubernetes.io/projected/bc83e08c-8b67-47b0-834d-bbb30387a45b-kube-api-access-2pkqf\") pod \"opendatahub-operator-controller-manager-5b8f94678b-97n94\" (UID: \"bc83e08c-8b67-47b0-834d-bbb30387a45b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-97n94" Oct 30 23:12:24.532572 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.532516 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bc83e08c-8b67-47b0-834d-bbb30387a45b-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b8f94678b-97n94\" (UID: \"bc83e08c-8b67-47b0-834d-bbb30387a45b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-97n94" Oct 30 23:12:24.532607 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.532570 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc83e08c-8b67-47b0-834d-bbb30387a45b-tmp\") pod \"opendatahub-operator-controller-manager-5b8f94678b-97n94\" (UID: \"bc83e08c-8b67-47b0-834d-bbb30387a45b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-97n94" Oct 30 23:12:24.572010 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.571945 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 23:12:24.572010 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.571957 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 23:12:24.572010 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.572005 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 23:12:24.572226 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.572010 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 23:12:24.572226 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.572026 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 23:12:24.572226 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.572030 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-zt2dv\"" Oct 30 23:12:24.584635 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.584613 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-97n94"] Oct 30 23:12:24.594230 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.594207 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:24.633448 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.633396 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bc83e08c-8b67-47b0-834d-bbb30387a45b-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b8f94678b-97n94\" (UID: \"bc83e08c-8b67-47b0-834d-bbb30387a45b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-97n94" Oct 30 23:12:24.633939 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.633447 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc83e08c-8b67-47b0-834d-bbb30387a45b-tmp\") pod \"opendatahub-operator-controller-manager-5b8f94678b-97n94\" (UID: \"bc83e08c-8b67-47b0-834d-bbb30387a45b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-97n94" Oct 30 23:12:24.633939 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.633505 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bc83e08c-8b67-47b0-834d-bbb30387a45b-manifests\") pod \"opendatahub-operator-controller-manager-5b8f94678b-97n94\" (UID: \"bc83e08c-8b67-47b0-834d-bbb30387a45b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-97n94" Oct 30 23:12:24.633939 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.633542 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bc83e08c-8b67-47b0-834d-bbb30387a45b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b8f94678b-97n94\" (UID: \"bc83e08c-8b67-47b0-834d-bbb30387a45b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-97n94" Oct 30 23:12:24.633939 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.633617 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bc83e08c-8b67-47b0-834d-bbb30387a45b-metrics-cert\") pod \"opendatahub-operator-controller-manager-5b8f94678b-97n94\" (UID: \"bc83e08c-8b67-47b0-834d-bbb30387a45b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-97n94" Oct 30 23:12:24.633939 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.633647 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pkqf\" (UniqueName: \"kubernetes.io/projected/bc83e08c-8b67-47b0-834d-bbb30387a45b-kube-api-access-2pkqf\") pod \"opendatahub-operator-controller-manager-5b8f94678b-97n94\" (UID: \"bc83e08c-8b67-47b0-834d-bbb30387a45b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-97n94" Oct 30 23:12:24.634318 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.634162 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bc83e08c-8b67-47b0-834d-bbb30387a45b-manifests\") pod \"opendatahub-operator-controller-manager-5b8f94678b-97n94\" (UID: \"bc83e08c-8b67-47b0-834d-bbb30387a45b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-97n94" Oct 30 23:12:24.636610 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.636587 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bc83e08c-8b67-47b0-834d-bbb30387a45b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b8f94678b-97n94\" (UID: \"bc83e08c-8b67-47b0-834d-bbb30387a45b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-97n94" Oct 30 23:12:24.636747 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.636644 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc83e08c-8b67-47b0-834d-bbb30387a45b-tmp\") pod \"opendatahub-operator-controller-manager-5b8f94678b-97n94\" (UID: \"bc83e08c-8b67-47b0-834d-bbb30387a45b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-97n94" Oct 30 23:12:24.636852 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.636833 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bc83e08c-8b67-47b0-834d-bbb30387a45b-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b8f94678b-97n94\" (UID: \"bc83e08c-8b67-47b0-834d-bbb30387a45b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-97n94" Oct 30 23:12:24.637091 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.637072 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bc83e08c-8b67-47b0-834d-bbb30387a45b-metrics-cert\") pod \"opendatahub-operator-controller-manager-5b8f94678b-97n94\" (UID: \"bc83e08c-8b67-47b0-834d-bbb30387a45b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-97n94" Oct 30 23:12:24.692517 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.692481 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pkqf\" (UniqueName: \"kubernetes.io/projected/bc83e08c-8b67-47b0-834d-bbb30387a45b-kube-api-access-2pkqf\") pod \"opendatahub-operator-controller-manager-5b8f94678b-97n94\" (UID: \"bc83e08c-8b67-47b0-834d-bbb30387a45b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-97n94" Oct 30 23:12:24.780363 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.780336 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-97n94" Oct 30 23:12:24.826468 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:24.825470 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:25.104501 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:25.104468 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:25.272122 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:25.272086 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-97n94"] Oct 30 23:12:25.275032 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:12:25.275005 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc83e08c_8b67_47b0_834d_bbb30387a45b.slice/crio-638810b911b18711ae1b30c9b64257324e3d1d29b79b7ca87065bca6178c3a55 WatchSource:0}: Error finding container 638810b911b18711ae1b30c9b64257324e3d1d29b79b7ca87065bca6178c3a55: Status 404 returned error can't find the container with id 638810b911b18711ae1b30c9b64257324e3d1d29b79b7ca87065bca6178c3a55 Oct 30 23:12:26.036636 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:26.036605 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-97n94" event={"ID":"bc83e08c-8b67-47b0-834d-bbb30387a45b","Type":"ContainerStarted","Data":"638810b911b18711ae1b30c9b64257324e3d1d29b79b7ca87065bca6178c3a55"} Oct 30 23:12:27.273360 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:27.273327 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:27.538946 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:27.538868 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:29.053716 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:29.053637 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-97n94" event={"ID":"bc83e08c-8b67-47b0-834d-bbb30387a45b","Type":"ContainerStarted","Data":"c98848732ec1831d0cc02e5795ca81420cb4923e447bb6af5c08b338fdac4ef7"} Oct 30 23:12:30.059403 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:30.059365 4952 generic.go:356] "Generic (PLEG): container finished" podID="bc83e08c-8b67-47b0-834d-bbb30387a45b" containerID="c98848732ec1831d0cc02e5795ca81420cb4923e447bb6af5c08b338fdac4ef7" exitCode=0 Oct 30 23:12:30.059794 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:30.059445 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-97n94" event={"ID":"bc83e08c-8b67-47b0-834d-bbb30387a45b","Type":"ContainerDied","Data":"c98848732ec1831d0cc02e5795ca81420cb4923e447bb6af5c08b338fdac4ef7"} Oct 30 23:12:31.065765 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:31.065725 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-97n94" event={"ID":"bc83e08c-8b67-47b0-834d-bbb30387a45b","Type":"ContainerStarted","Data":"8984370ad06f6401a1353664fece6b91e09095fc946a3f655fdfd4d30aa94211"} Oct 30 23:12:31.066155 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:31.065825 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-97n94" Oct 30 23:12:31.266479 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:31.266374 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-97n94" podStartSLOduration=2.5475296739999997 podStartE2EDuration="7.266362101s" totalImagesPullingTime="4.279045718s" totalInitContainerRuntime="439.786709ms" isStatefulPod=false podCreationTimestamp="2026-10-30 23:12:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:12:31.255524083 +0000 UTC m=+370.399354307" watchObservedRunningTime="2026-10-30 23:12:31.266362101 +0000 UTC m=+370.410192305" Oct 30 23:12:31.433198 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:31.433170 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ttgh9"] Oct 30 23:12:31.436445 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:31.436430 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ttgh9" Oct 30 23:12:31.492259 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:31.492225 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 23:12:31.492388 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:31.492359 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-j8zx2\"" Oct 30 23:12:31.494892 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:31.494874 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:12:31.500371 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:31.500348 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ttgh9"] Oct 30 23:12:31.593238 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:31.593212 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fede345-66de-4ee3-9d65-9c678c392025-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ttgh9\" (UID: \"8fede345-66de-4ee3-9d65-9c678c392025\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ttgh9" Oct 30 23:12:31.593380 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:31.593252 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gbxm\" (UniqueName: \"kubernetes.io/projected/8fede345-66de-4ee3-9d65-9c678c392025-kube-api-access-6gbxm\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ttgh9\" (UID: \"8fede345-66de-4ee3-9d65-9c678c392025\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ttgh9" Oct 30 23:12:31.693782 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:31.693756 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fede345-66de-4ee3-9d65-9c678c392025-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ttgh9\" (UID: \"8fede345-66de-4ee3-9d65-9c678c392025\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ttgh9" Oct 30 23:12:31.693942 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:31.693793 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gbxm\" (UniqueName: \"kubernetes.io/projected/8fede345-66de-4ee3-9d65-9c678c392025-kube-api-access-6gbxm\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ttgh9\" (UID: \"8fede345-66de-4ee3-9d65-9c678c392025\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ttgh9" Oct 30 23:12:31.694239 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:31.694213 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fede345-66de-4ee3-9d65-9c678c392025-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ttgh9\" (UID: \"8fede345-66de-4ee3-9d65-9c678c392025\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ttgh9" Oct 30 23:12:31.750912 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:31.750885 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gbxm\" (UniqueName: \"kubernetes.io/projected/8fede345-66de-4ee3-9d65-9c678c392025-kube-api-access-6gbxm\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ttgh9\" (UID: \"8fede345-66de-4ee3-9d65-9c678c392025\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ttgh9" Oct 30 23:12:32.048347 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:32.048267 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ttgh9" Oct 30 23:12:32.522827 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:32.522801 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ttgh9"] Oct 30 23:12:32.524882 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:12:32.524853 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fede345_66de_4ee3_9d65_9c678c392025.slice/crio-ccff69aff41766868c6a84e7831deb5c57f496ca71e9852ca76344f6b79bc1fd WatchSource:0}: Error finding container ccff69aff41766868c6a84e7831deb5c57f496ca71e9852ca76344f6b79bc1fd: Status 404 returned error can't find the container with id ccff69aff41766868c6a84e7831deb5c57f496ca71e9852ca76344f6b79bc1fd Oct 30 23:12:32.917656 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:32.917626 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:33.076737 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:33.076703 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ttgh9" event={"ID":"8fede345-66de-4ee3-9d65-9c678c392025","Type":"ContainerStarted","Data":"ccff69aff41766868c6a84e7831deb5c57f496ca71e9852ca76344f6b79bc1fd"} Oct 30 23:12:33.196484 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:33.196409 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:33.442841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:33.442803 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:34.668565 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:34.668532 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:34.961028 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:34.960944 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:36.093735 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:36.093704 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ttgh9" event={"ID":"8fede345-66de-4ee3-9d65-9c678c392025","Type":"ContainerStarted","Data":"48a7062507c0398ebec7a2c652f493a2394bff5d56bff1d339d8f148f37b32d2"} Oct 30 23:12:36.322113 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:36.322057 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ttgh9" podStartSLOduration=2.531485687 podStartE2EDuration="5.32201835s" totalImagesPullingTime="2.790532663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:12:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:12:36.271691416 +0000 UTC m=+375.415521620" watchObservedRunningTime="2026-10-30 23:12:36.32201835 +0000 UTC m=+375.465848555" Oct 30 23:12:37.952873 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:37.952838 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:38.233907 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:38.233835 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:38.732889 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:38.732854 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-pkx96"] Oct 30 23:12:38.736430 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:38.736415 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-pkx96" Oct 30 23:12:38.792592 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:38.792566 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 23:12:38.792592 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:38.792566 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 23:12:38.842614 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:38.842589 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-nrjrr\"" Oct 30 23:12:38.852970 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:38.852943 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkxkh\" (UniqueName: \"kubernetes.io/projected/8f8b6a0c-282f-45f4-a2ea-500a5cf81dd4-kube-api-access-tkxkh\") pod \"authorino-operator-7b7bb4bc77-pkx96\" (UID: \"8f8b6a0c-282f-45f4-a2ea-500a5cf81dd4\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-pkx96" Oct 30 23:12:38.864457 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:38.864436 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-pkx96"] Oct 30 23:12:38.953515 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:38.953489 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkxkh\" (UniqueName: \"kubernetes.io/projected/8f8b6a0c-282f-45f4-a2ea-500a5cf81dd4-kube-api-access-tkxkh\") pod \"authorino-operator-7b7bb4bc77-pkx96\" (UID: \"8f8b6a0c-282f-45f4-a2ea-500a5cf81dd4\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-pkx96" Oct 30 23:12:39.013813 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:39.013754 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkxkh\" (UniqueName: \"kubernetes.io/projected/8f8b6a0c-282f-45f4-a2ea-500a5cf81dd4-kube-api-access-tkxkh\") pod \"authorino-operator-7b7bb4bc77-pkx96\" (UID: \"8f8b6a0c-282f-45f4-a2ea-500a5cf81dd4\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-pkx96" Oct 30 23:12:39.048705 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:39.048677 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-pkx96" Oct 30 23:12:39.456999 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:39.456975 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-pkx96"] Oct 30 23:12:39.458804 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:12:39.458777 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8b6a0c_282f_45f4_a2ea_500a5cf81dd4.slice/crio-ac3b80e677ec8be44f428a5fa3d1c05fccdf9ec99c3852f170ebde0dcccf1293 WatchSource:0}: Error finding container ac3b80e677ec8be44f428a5fa3d1c05fccdf9ec99c3852f170ebde0dcccf1293: Status 404 returned error can't find the container with id ac3b80e677ec8be44f428a5fa3d1c05fccdf9ec99c3852f170ebde0dcccf1293 Oct 30 23:12:40.053168 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:40.053135 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:40.120029 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:40.119992 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-pkx96" event={"ID":"8f8b6a0c-282f-45f4-a2ea-500a5cf81dd4","Type":"ContainerStarted","Data":"ac3b80e677ec8be44f428a5fa3d1c05fccdf9ec99c3852f170ebde0dcccf1293"} Oct 30 23:12:40.356462 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:40.356430 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:40.623026 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:40.622954 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:40.956146 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:40.956071 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:42.073963 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:42.073934 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-97n94" Oct 30 23:12:42.262252 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:42.262218 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-pmgl5"] Oct 30 23:12:42.282637 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:42.282606 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pmgl5" Oct 30 23:12:42.338373 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:42.338344 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 23:12:42.338816 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:42.338779 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-h2vnw\"" Oct 30 23:12:42.381328 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:42.381295 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnxpr\" (UniqueName: \"kubernetes.io/projected/5f95de50-95f0-4e61-a2a0-deacaece6891-kube-api-access-mnxpr\") pod \"dns-operator-controller-manager-8557cbdb84-pmgl5\" (UID: \"5f95de50-95f0-4e61-a2a0-deacaece6891\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pmgl5" Oct 30 23:12:42.398802 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:42.398768 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-pmgl5"] Oct 30 23:12:42.482085 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:42.482022 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnxpr\" (UniqueName: \"kubernetes.io/projected/5f95de50-95f0-4e61-a2a0-deacaece6891-kube-api-access-mnxpr\") pod \"dns-operator-controller-manager-8557cbdb84-pmgl5\" (UID: \"5f95de50-95f0-4e61-a2a0-deacaece6891\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pmgl5" Oct 30 23:12:42.551560 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:42.551528 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnxpr\" (UniqueName: \"kubernetes.io/projected/5f95de50-95f0-4e61-a2a0-deacaece6891-kube-api-access-mnxpr\") pod \"dns-operator-controller-manager-8557cbdb84-pmgl5\" (UID: \"5f95de50-95f0-4e61-a2a0-deacaece6891\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pmgl5" Oct 30 23:12:42.613300 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:42.613222 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pmgl5" Oct 30 23:12:42.752261 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:42.752230 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:43.078592 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:43.078566 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-pmgl5"] Oct 30 23:12:43.080419 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:12:43.080392 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f95de50_95f0_4e61_a2a0_deacaece6891.slice/crio-3d46c8e4968b7fb0a301c0e481e899fd4c69ad3c2fa4245e3803b83178993d57 WatchSource:0}: Error finding container 3d46c8e4968b7fb0a301c0e481e899fd4c69ad3c2fa4245e3803b83178993d57: Status 404 returned error can't find the container with id 3d46c8e4968b7fb0a301c0e481e899fd4c69ad3c2fa4245e3803b83178993d57 Oct 30 23:12:43.137513 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:43.137482 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pmgl5" event={"ID":"5f95de50-95f0-4e61-a2a0-deacaece6891","Type":"ContainerStarted","Data":"3d46c8e4968b7fb0a301c0e481e899fd4c69ad3c2fa4245e3803b83178993d57"} Oct 30 23:12:43.836644 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:43.836613 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:44.145104 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:44.145065 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-pkx96" event={"ID":"8f8b6a0c-282f-45f4-a2ea-500a5cf81dd4","Type":"ContainerStarted","Data":"2137ac9f60cda4b14667bc56746b5568de9301cfb2447eeed752ca589f84134e"} Oct 30 23:12:44.145662 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:44.145197 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-pkx96" Oct 30 23:12:44.273642 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:44.273585 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-pkx96" podStartSLOduration=2.044764843 podStartE2EDuration="6.273571687s" totalImagesPullingTime="4.228806844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:12:44.273152262 +0000 UTC m=+383.416982469" watchObservedRunningTime="2026-10-30 23:12:44.273571687 +0000 UTC m=+383.417401892" Oct 30 23:12:45.246858 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:45.246820 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:45.864248 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:45.864225 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:46.156001 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:46.155921 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:46.158900 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:46.158875 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pmgl5" event={"ID":"5f95de50-95f0-4e61-a2a0-deacaece6891","Type":"ContainerStarted","Data":"1eb3638a04b7a75a507989153e503e15a47c458f04fb44cb70b4690c5aafaf81"} Oct 30 23:12:46.159063 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:46.158963 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pmgl5" Oct 30 23:12:46.276676 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:46.276619 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pmgl5" podStartSLOduration=1.501693838 podStartE2EDuration="4.276606108s" totalImagesPullingTime="2.77491227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:12:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:12:46.275239118 +0000 UTC m=+385.419069337" watchObservedRunningTime="2026-10-30 23:12:46.276606108 +0000 UTC m=+385.420436312" Oct 30 23:12:46.615180 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:46.615147 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:46.933103 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:46.933013 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:47.020552 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:47.020323 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-9hxjd"] Oct 30 23:12:47.024932 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:47.024901 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9hxjd" Oct 30 23:12:47.118822 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:47.118790 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j6jk\" (UniqueName: \"kubernetes.io/projected/e6751225-439f-4088-a474-183cd2d75baa-kube-api-access-9j6jk\") pod \"limitador-operator-controller-manager-5778bb4449-9hxjd\" (UID: \"e6751225-439f-4088-a474-183cd2d75baa\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9hxjd" Oct 30 23:12:47.130688 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:47.130656 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-9hxjd"] Oct 30 23:12:47.164837 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:47.164811 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-4j5jl\"" Oct 30 23:12:47.219964 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:47.219889 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j6jk\" (UniqueName: \"kubernetes.io/projected/e6751225-439f-4088-a474-183cd2d75baa-kube-api-access-9j6jk\") pod \"limitador-operator-controller-manager-5778bb4449-9hxjd\" (UID: \"e6751225-439f-4088-a474-183cd2d75baa\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9hxjd" Oct 30 23:12:47.296740 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:47.296713 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j6jk\" (UniqueName: \"kubernetes.io/projected/e6751225-439f-4088-a474-183cd2d75baa-kube-api-access-9j6jk\") pod \"limitador-operator-controller-manager-5778bb4449-9hxjd\" (UID: \"e6751225-439f-4088-a474-183cd2d75baa\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9hxjd" Oct 30 23:12:47.336553 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:47.336527 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9hxjd" Oct 30 23:12:47.696381 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:47.696345 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:47.779373 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:47.779342 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-9hxjd"] Oct 30 23:12:47.782852 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:12:47.782824 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6751225_439f_4088_a474_183cd2d75baa.slice/crio-15fa2924c0993294698a91702c8ace1984278d1743fb6061fce2cd0fc1ea9dc8 WatchSource:0}: Error finding container 15fa2924c0993294698a91702c8ace1984278d1743fb6061fce2cd0fc1ea9dc8: Status 404 returned error can't find the container with id 15fa2924c0993294698a91702c8ace1984278d1743fb6061fce2cd0fc1ea9dc8 Oct 30 23:12:48.171574 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:48.171543 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9hxjd" event={"ID":"e6751225-439f-4088-a474-183cd2d75baa","Type":"ContainerStarted","Data":"15fa2924c0993294698a91702c8ace1984278d1743fb6061fce2cd0fc1ea9dc8"} Oct 30 23:12:48.224769 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:48.224741 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:48.545477 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:48.545394 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:49.303327 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:49.303293 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:51.141679 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:12:51.141642 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:12:51.188792 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:51.188757 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9hxjd" event={"ID":"e6751225-439f-4088-a474-183cd2d75baa","Type":"ContainerStarted","Data":"ed1b9e722d9dbfe689922a53e4eb7ae868479a7f790e2174c3ca15f22e4ffa2d"} Oct 30 23:12:51.188965 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:51.188818 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9hxjd" Oct 30 23:12:51.306812 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:51.306763 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9hxjd" podStartSLOduration=2.76067654 podStartE2EDuration="5.306748687s" totalImagesPullingTime="2.546072147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:12:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:12:51.306258811 +0000 UTC m=+390.450089036" watchObservedRunningTime="2026-10-30 23:12:51.306748687 +0000 UTC m=+390.450578892" Oct 30 23:12:54.029310 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:54.029273 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-gl24g"] Oct 30 23:12:54.030695 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:54.030661 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:54.033095 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:54.033079 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-gl24g" Oct 30 23:12:54.076792 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:54.076766 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zlr7\" (UniqueName: \"kubernetes.io/projected/9581e345-633d-4e29-beea-cc64366ef77f-kube-api-access-8zlr7\") pod \"kuadrant-operator-controller-manager-cf8469d7b-gl24g\" (UID: \"9581e345-633d-4e29-beea-cc64366ef77f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-gl24g" Oct 30 23:12:54.076943 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:54.076804 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9581e345-633d-4e29-beea-cc64366ef77f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-gl24g\" (UID: \"9581e345-633d-4e29-beea-cc64366ef77f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-gl24g" Oct 30 23:12:54.080622 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:54.080598 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-gl24g"] Oct 30 23:12:54.128068 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:54.128021 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rsddc\"" Oct 30 23:12:54.178312 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:54.178284 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zlr7\" (UniqueName: \"kubernetes.io/projected/9581e345-633d-4e29-beea-cc64366ef77f-kube-api-access-8zlr7\") pod \"kuadrant-operator-controller-manager-cf8469d7b-gl24g\" (UID: \"9581e345-633d-4e29-beea-cc64366ef77f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-gl24g" Oct 30 23:12:54.178419 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:54.178318 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9581e345-633d-4e29-beea-cc64366ef77f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-gl24g\" (UID: \"9581e345-633d-4e29-beea-cc64366ef77f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-gl24g" Oct 30 23:12:54.178687 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:54.178671 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9581e345-633d-4e29-beea-cc64366ef77f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-gl24g\" (UID: \"9581e345-633d-4e29-beea-cc64366ef77f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-gl24g" Oct 30 23:12:54.235395 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:54.235369 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zlr7\" (UniqueName: \"kubernetes.io/projected/9581e345-633d-4e29-beea-cc64366ef77f-kube-api-access-8zlr7\") pod \"kuadrant-operator-controller-manager-cf8469d7b-gl24g\" (UID: \"9581e345-633d-4e29-beea-cc64366ef77f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-gl24g" Oct 30 23:12:54.271686 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:54.271661 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:54.345308 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:54.345284 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-gl24g" Oct 30 23:12:54.580154 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:54.580126 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:54.833510 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:54.833487 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-gl24g"] Oct 30 23:12:54.835121 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:12:54.835096 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9581e345_633d_4e29_beea_cc64366ef77f.slice/crio-aa80b6063e49d8c86de63f09cc8437062c910458d95a18f3ad61ed6f23fd0c30 WatchSource:0}: Error finding container aa80b6063e49d8c86de63f09cc8437062c910458d95a18f3ad61ed6f23fd0c30: Status 404 returned error can't find the container with id aa80b6063e49d8c86de63f09cc8437062c910458d95a18f3ad61ed6f23fd0c30 Oct 30 23:12:55.153934 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:55.153907 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-pkx96" Oct 30 23:12:55.209274 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:55.209244 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-gl24g" event={"ID":"9581e345-633d-4e29-beea-cc64366ef77f","Type":"ContainerStarted","Data":"aa80b6063e49d8c86de63f09cc8437062c910458d95a18f3ad61ed6f23fd0c30"} Oct 30 23:12:56.056694 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:56.056653 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:56.389965 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:56.389934 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:56.640266 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:56.640180 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:56.898155 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:56.898082 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:57.167892 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:57.167808 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pmgl5" Oct 30 23:12:58.907239 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:58.907198 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:59.152014 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:59.151951 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:12:59.637076 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:12:59.637028 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:13:00.240821 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:00.240741 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-gl24g" event={"ID":"9581e345-633d-4e29-beea-cc64366ef77f","Type":"ContainerStarted","Data":"65e648e6c2a2e8fe252ad0b386d7d29d2761a98eda8bd1d101ff74fe5be48a6d"} Oct 30 23:13:00.241310 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:00.240878 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-gl24g" Oct 30 23:13:00.462122 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:00.462070 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-gl24g" podStartSLOduration=2.381273486 podStartE2EDuration="7.462054657s" totalImagesPullingTime="5.080781171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:12:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:13:00.410498879 +0000 UTC m=+399.554329104" watchObservedRunningTime="2026-10-30 23:13:00.462054657 +0000 UTC m=+399.605884859" Oct 30 23:13:01.335815 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:01.335781 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:13:01.597246 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:01.597214 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:13:01.835271 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:01.835236 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr"] Oct 30 23:13:01.838873 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:01.838854 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" Oct 30 23:13:01.882451 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:01.882371 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:13:01.891264 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:01.891244 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-4kr79\"" Oct 30 23:13:01.900092 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:01.900066 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr"] Oct 30 23:13:01.949063 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:01.949014 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-56c7694f852x7dr\" (UID: \"ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" Oct 30 23:13:01.949063 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:01.949059 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sdv6\" (UniqueName: \"kubernetes.io/projected/ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe-kube-api-access-4sdv6\") pod \"data-science-gateway-data-science-gateway-class-56c7694f852x7dr\" (UID: \"ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" Oct 30 23:13:01.949261 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:01.949163 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-56c7694f852x7dr\" (UID: \"ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" Oct 30 23:13:01.949261 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:01.949221 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-56c7694f852x7dr\" (UID: \"ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" Oct 30 23:13:01.949261 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:01.949246 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-56c7694f852x7dr\" (UID: \"ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" Oct 30 23:13:01.949357 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:01.949301 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe-istio-data\") pod \"data-science-gateway-data-science-gateway-class-56c7694f852x7dr\" (UID: \"ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" Oct 30 23:13:01.949357 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:01.949334 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-56c7694f852x7dr\" (UID: \"ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" Oct 30 23:13:01.949418 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:01.949359 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe-istio-token\") pod \"data-science-gateway-data-science-gateway-class-56c7694f852x7dr\" (UID: \"ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" Oct 30 23:13:01.949451 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:01.949443 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-56c7694f852x7dr\" (UID: \"ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" Oct 30 23:13:02.050362 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:02.050327 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe-istio-data\") pod \"data-science-gateway-data-science-gateway-class-56c7694f852x7dr\" (UID: \"ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" Oct 30 23:13:02.050528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:02.050376 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-56c7694f852x7dr\" (UID: \"ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" Oct 30 23:13:02.050528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:02.050497 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe-istio-token\") pod \"data-science-gateway-data-science-gateway-class-56c7694f852x7dr\" (UID: \"ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" Oct 30 23:13:02.050626 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:02.050610 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-56c7694f852x7dr\" (UID: \"ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" Oct 30 23:13:02.050698 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:02.050677 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-56c7694f852x7dr\" (UID: \"ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" Oct 30 23:13:02.050758 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:02.050723 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sdv6\" (UniqueName: \"kubernetes.io/projected/ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe-kube-api-access-4sdv6\") pod \"data-science-gateway-data-science-gateway-class-56c7694f852x7dr\" (UID: \"ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" Oct 30 23:13:02.050810 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:02.050755 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-56c7694f852x7dr\" (UID: \"ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" Oct 30 23:13:02.050810 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:02.050794 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-56c7694f852x7dr\" (UID: \"ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" Oct 30 23:13:02.050810 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:02.050801 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-56c7694f852x7dr\" (UID: \"ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" Oct 30 23:13:02.050996 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:02.050756 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe-istio-data\") pod \"data-science-gateway-data-science-gateway-class-56c7694f852x7dr\" (UID: \"ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" Oct 30 23:13:02.050996 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:02.050845 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-56c7694f852x7dr\" (UID: \"ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" Oct 30 23:13:02.050996 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:02.050950 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-56c7694f852x7dr\" (UID: \"ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" Oct 30 23:13:02.051185 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:02.051068 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-56c7694f852x7dr\" (UID: \"ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" Oct 30 23:13:02.051420 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:02.051401 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-56c7694f852x7dr\" (UID: \"ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" Oct 30 23:13:02.053128 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:02.053104 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-56c7694f852x7dr\" (UID: \"ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" Oct 30 23:13:02.053457 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:02.053436 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-56c7694f852x7dr\" (UID: \"ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" Oct 30 23:13:02.113366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:02.113334 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe-istio-token\") pod \"data-science-gateway-data-science-gateway-class-56c7694f852x7dr\" (UID: \"ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" Oct 30 23:13:02.113506 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:02.113454 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sdv6\" (UniqueName: \"kubernetes.io/projected/ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe-kube-api-access-4sdv6\") pod \"data-science-gateway-data-science-gateway-class-56c7694f852x7dr\" (UID: \"ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" Oct 30 23:13:02.152146 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:02.152091 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" Oct 30 23:13:02.160865 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:02.160837 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:13:02.197227 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:02.196443 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9hxjd" Oct 30 23:13:02.474325 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:02.474245 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:13:02.589164 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:02.589137 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr"] Oct 30 23:13:02.590232 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:13:02.590205 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad91d15d_ceb9_464a_8a8a_aa9bf61eb7fe.slice/crio-d58166dd1af212e415e6cbbccede54abe59a3c8db7d562485d6631c41b35ac4d WatchSource:0}: Error finding container d58166dd1af212e415e6cbbccede54abe59a3c8db7d562485d6631c41b35ac4d: Status 404 returned error can't find the container with id d58166dd1af212e415e6cbbccede54abe59a3c8db7d562485d6631c41b35ac4d Oct 30 23:13:02.592582 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:02.592544 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:13:02.592675 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:02.592635 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:13:02.592717 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:02.592679 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:13:03.258324 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:03.258284 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" event={"ID":"ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe","Type":"ContainerStarted","Data":"891c3ddae54503175e1ba4d140aedf00cc9bc118d9b5b45d223fd6b97a080a23"} Oct 30 23:13:03.258516 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:03.258351 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" event={"ID":"ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe","Type":"ContainerStarted","Data":"d58166dd1af212e415e6cbbccede54abe59a3c8db7d562485d6631c41b35ac4d"} Oct 30 23:13:03.476336 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:03.476283 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" podStartSLOduration=2.476268026 podStartE2EDuration="2.476268026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:13:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:13:03.42590088 +0000 UTC m=+402.569731084" watchObservedRunningTime="2026-10-30 23:13:03.476268026 +0000 UTC m=+402.620098230" Oct 30 23:13:04.153266 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:04.153235 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" Oct 30 23:13:04.157820 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:04.157798 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" Oct 30 23:13:04.263057 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:04.263000 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" Oct 30 23:13:04.264165 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:04.264142 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f852x7dr" Oct 30 23:13:04.416474 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:04.416406 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:13:04.647201 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:04.647169 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:13:07.701811 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:07.701779 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:13:07.928681 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:07.928648 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:13:08.487307 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:08.487275 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-75gxr"] Oct 30 23:13:08.504855 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:08.504825 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-75gxr" Oct 30 23:13:08.521094 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:08.521066 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:13:08.556599 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:08.556577 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 23:13:08.557353 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:08.557334 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-75gxr"] Oct 30 23:13:08.602848 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:08.602825 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-62hmr\"" Oct 30 23:13:08.606130 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:08.606114 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 23:13:08.610530 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:08.610507 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb7db2c7-13a2-44b6-83f3-b5966cff4343-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-75gxr\" (UID: \"fb7db2c7-13a2-44b6-83f3-b5966cff4343\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-75gxr" Oct 30 23:13:08.610615 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:08.610543 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frjr8\" (UniqueName: \"kubernetes.io/projected/fb7db2c7-13a2-44b6-83f3-b5966cff4343-kube-api-access-frjr8\") pod \"kuadrant-console-plugin-66c4fb575d-75gxr\" (UID: \"fb7db2c7-13a2-44b6-83f3-b5966cff4343\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-75gxr" Oct 30 23:13:08.610615 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:08.610584 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fb7db2c7-13a2-44b6-83f3-b5966cff4343-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-75gxr\" (UID: \"fb7db2c7-13a2-44b6-83f3-b5966cff4343\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-75gxr" Oct 30 23:13:08.711330 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:08.711302 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fb7db2c7-13a2-44b6-83f3-b5966cff4343-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-75gxr\" (UID: \"fb7db2c7-13a2-44b6-83f3-b5966cff4343\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-75gxr" Oct 30 23:13:08.711771 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:08.711369 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb7db2c7-13a2-44b6-83f3-b5966cff4343-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-75gxr\" (UID: \"fb7db2c7-13a2-44b6-83f3-b5966cff4343\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-75gxr" Oct 30 23:13:08.711771 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:08.711402 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frjr8\" (UniqueName: \"kubernetes.io/projected/fb7db2c7-13a2-44b6-83f3-b5966cff4343-kube-api-access-frjr8\") pod \"kuadrant-console-plugin-66c4fb575d-75gxr\" (UID: \"fb7db2c7-13a2-44b6-83f3-b5966cff4343\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-75gxr" Oct 30 23:13:08.712200 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:08.712177 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fb7db2c7-13a2-44b6-83f3-b5966cff4343-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-75gxr\" (UID: \"fb7db2c7-13a2-44b6-83f3-b5966cff4343\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-75gxr" Oct 30 23:13:08.714024 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:08.713998 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb7db2c7-13a2-44b6-83f3-b5966cff4343-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-75gxr\" (UID: \"fb7db2c7-13a2-44b6-83f3-b5966cff4343\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-75gxr" Oct 30 23:13:08.758702 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:08.758627 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:13:08.769850 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:08.769821 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frjr8\" (UniqueName: \"kubernetes.io/projected/fb7db2c7-13a2-44b6-83f3-b5966cff4343-kube-api-access-frjr8\") pod \"kuadrant-console-plugin-66c4fb575d-75gxr\" (UID: \"fb7db2c7-13a2-44b6-83f3-b5966cff4343\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-75gxr" Oct 30 23:13:08.827307 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:08.827280 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-75gxr" Oct 30 23:13:08.985633 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:08.985600 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:13:09.213705 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:09.213681 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-75gxr"] Oct 30 23:13:09.215393 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:13:09.215368 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb7db2c7_13a2_44b6_83f3_b5966cff4343.slice/crio-57fb635c2115182dd91dadc7a30625ce7f93d376a73d6a52f5b96614740f1264 WatchSource:0}: Error finding container 57fb635c2115182dd91dadc7a30625ce7f93d376a73d6a52f5b96614740f1264: Status 404 returned error can't find the container with id 57fb635c2115182dd91dadc7a30625ce7f93d376a73d6a52f5b96614740f1264 Oct 30 23:13:09.240758 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:09.240730 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:13:09.289154 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:09.289128 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-75gxr" event={"ID":"fb7db2c7-13a2-44b6-83f3-b5966cff4343","Type":"ContainerStarted","Data":"57fb635c2115182dd91dadc7a30625ce7f93d376a73d6a52f5b96614740f1264"} Oct 30 23:13:09.542190 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:09.542086 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:13:10.014463 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:10.014429 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-mpr8c"] Oct 30 23:13:10.019351 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:10.019331 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-mpr8c" Oct 30 23:13:10.071742 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:10.071714 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-mpr8c"] Oct 30 23:13:10.119183 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:10.119149 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kntdc\"" Oct 30 23:13:10.121526 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:10.121496 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwppc\" (UniqueName: \"kubernetes.io/projected/eea7057e-988a-456f-8b89-d909848d8bb7-kube-api-access-mwppc\") pod \"authorino-57f7f8659f-mpr8c\" (UID: \"eea7057e-988a-456f-8b89-d909848d8bb7\") " pod="kuadrant-system/authorino-57f7f8659f-mpr8c" Oct 30 23:13:10.222591 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:10.222563 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwppc\" (UniqueName: \"kubernetes.io/projected/eea7057e-988a-456f-8b89-d909848d8bb7-kube-api-access-mwppc\") pod \"authorino-57f7f8659f-mpr8c\" (UID: \"eea7057e-988a-456f-8b89-d909848d8bb7\") " pod="kuadrant-system/authorino-57f7f8659f-mpr8c" Oct 30 23:13:10.279870 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:10.279812 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwppc\" (UniqueName: \"kubernetes.io/projected/eea7057e-988a-456f-8b89-d909848d8bb7-kube-api-access-mwppc\") pod \"authorino-57f7f8659f-mpr8c\" (UID: \"eea7057e-988a-456f-8b89-d909848d8bb7\") " pod="kuadrant-system/authorino-57f7f8659f-mpr8c" Oct 30 23:13:10.291572 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:10.291543 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:13:10.331963 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:10.331937 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-mpr8c" Oct 30 23:13:10.516314 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:10.516282 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:13:10.700668 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:10.700644 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-mpr8c"] Oct 30 23:13:10.701901 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:13:10.701873 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeea7057e_988a_456f_8b89_d909848d8bb7.slice/crio-07bb1af5c4762d0aeef145e8c0302e901f9eadeceb58e0c9fa9bac05594a0015 WatchSource:0}: Error finding container 07bb1af5c4762d0aeef145e8c0302e901f9eadeceb58e0c9fa9bac05594a0015: Status 404 returned error can't find the container with id 07bb1af5c4762d0aeef145e8c0302e901f9eadeceb58e0c9fa9bac05594a0015 Oct 30 23:13:11.248206 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:11.248157 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-gl24g" Oct 30 23:13:11.302725 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:11.302687 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-mpr8c" event={"ID":"eea7057e-988a-456f-8b89-d909848d8bb7","Type":"ContainerStarted","Data":"07bb1af5c4762d0aeef145e8c0302e901f9eadeceb58e0c9fa9bac05594a0015"} Oct 30 23:13:12.769720 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:12.769632 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:13:13.158519 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:13.158489 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qxr4b"] Oct 30 23:13:13.163464 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:13.163435 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-qxr4b" Oct 30 23:13:13.182565 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:13.182537 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:13:13.186387 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:13.186366 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-hj2mm" Oct 30 23:13:13.229901 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:13.229872 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qxr4b"] Oct 30 23:13:13.253324 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:13.253298 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 23:13:13.253705 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:13.253688 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bd4s\" (UniqueName: \"kubernetes.io/projected/fb5cde51-c849-4371-9e64-fbfa54e26e35-kube-api-access-7bd4s\") pod \"authorino-56c6c9ddd5-qxr4b\" (UID: \"fb5cde51-c849-4371-9e64-fbfa54e26e35\") " pod="kuadrant-system/authorino-56c6c9ddd5-qxr4b" Oct 30 23:13:13.253791 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:13.253735 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx8db\" (UniqueName: \"kubernetes.io/projected/52beb09a-acb7-4f24-a70b-195f1939ba32-kube-api-access-zx8db\") pod \"limitador-limitador-7c9b87d574-hj2mm\" (UID: \"52beb09a-acb7-4f24-a70b-195f1939ba32\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-hj2mm" Oct 30 23:13:13.253896 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:13.253868 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/52beb09a-acb7-4f24-a70b-195f1939ba32-config-file\") pod \"limitador-limitador-7c9b87d574-hj2mm\" (UID: \"52beb09a-acb7-4f24-a70b-195f1939ba32\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-hj2mm" Oct 30 23:13:13.266780 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:13.266751 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:13:13.354599 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:13.354563 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx8db\" (UniqueName: \"kubernetes.io/projected/52beb09a-acb7-4f24-a70b-195f1939ba32-kube-api-access-zx8db\") pod \"limitador-limitador-7c9b87d574-hj2mm\" (UID: \"52beb09a-acb7-4f24-a70b-195f1939ba32\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-hj2mm" Oct 30 23:13:13.354765 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:13.354617 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/52beb09a-acb7-4f24-a70b-195f1939ba32-config-file\") pod \"limitador-limitador-7c9b87d574-hj2mm\" (UID: \"52beb09a-acb7-4f24-a70b-195f1939ba32\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-hj2mm" Oct 30 23:13:13.354765 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:13.354673 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bd4s\" (UniqueName: \"kubernetes.io/projected/fb5cde51-c849-4371-9e64-fbfa54e26e35-kube-api-access-7bd4s\") pod \"authorino-56c6c9ddd5-qxr4b\" (UID: \"fb5cde51-c849-4371-9e64-fbfa54e26e35\") " pod="kuadrant-system/authorino-56c6c9ddd5-qxr4b" Oct 30 23:13:13.355760 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:13.355701 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/52beb09a-acb7-4f24-a70b-195f1939ba32-config-file\") pod \"limitador-limitador-7c9b87d574-hj2mm\" (UID: \"52beb09a-acb7-4f24-a70b-195f1939ba32\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-hj2mm" Oct 30 23:13:13.417374 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:13.417299 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bd4s\" (UniqueName: \"kubernetes.io/projected/fb5cde51-c849-4371-9e64-fbfa54e26e35-kube-api-access-7bd4s\") pod \"authorino-56c6c9ddd5-qxr4b\" (UID: \"fb5cde51-c849-4371-9e64-fbfa54e26e35\") " pod="kuadrant-system/authorino-56c6c9ddd5-qxr4b" Oct 30 23:13:13.417880 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:13.417860 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx8db\" (UniqueName: \"kubernetes.io/projected/52beb09a-acb7-4f24-a70b-195f1939ba32-kube-api-access-zx8db\") pod \"limitador-limitador-7c9b87d574-hj2mm\" (UID: \"52beb09a-acb7-4f24-a70b-195f1939ba32\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-hj2mm" Oct 30 23:13:13.467267 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:13.467224 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:13:13.480396 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:13.480367 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-qxr4b" Oct 30 23:13:13.502653 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:13.502626 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-hj2mm" Oct 30 23:13:13.893433 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:13.893402 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qxr4b"] Oct 30 23:13:13.895280 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:13:13.895240 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb5cde51_c849_4371_9e64_fbfa54e26e35.slice/crio-a3eaed066c0c51fee61c54d9ef088f8f26ff4ba25aff9add2c316cfd73eb1bcf WatchSource:0}: Error finding container a3eaed066c0c51fee61c54d9ef088f8f26ff4ba25aff9add2c316cfd73eb1bcf: Status 404 returned error can't find the container with id a3eaed066c0c51fee61c54d9ef088f8f26ff4ba25aff9add2c316cfd73eb1bcf Oct 30 23:13:13.934279 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:13.934257 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:13:13.935977 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:13:13.935949 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52beb09a_acb7_4f24_a70b_195f1939ba32.slice/crio-f755d4e48f8c684feacbdfa9719ba3e92a40d5642fc90fb8b2716ea1f4b3476c WatchSource:0}: Error finding container f755d4e48f8c684feacbdfa9719ba3e92a40d5642fc90fb8b2716ea1f4b3476c: Status 404 returned error can't find the container with id f755d4e48f8c684feacbdfa9719ba3e92a40d5642fc90fb8b2716ea1f4b3476c Oct 30 23:13:14.323774 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:14.323696 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-mpr8c" event={"ID":"eea7057e-988a-456f-8b89-d909848d8bb7","Type":"ContainerStarted","Data":"8d647df6ac3199154d1c75c81bd8383b2689082536e8e22cd598b6458cc48d94"} Oct 30 23:13:14.325342 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:14.325309 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-hj2mm" event={"ID":"52beb09a-acb7-4f24-a70b-195f1939ba32","Type":"ContainerStarted","Data":"f755d4e48f8c684feacbdfa9719ba3e92a40d5642fc90fb8b2716ea1f4b3476c"} Oct 30 23:13:14.326604 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:14.326570 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-qxr4b" event={"ID":"fb5cde51-c849-4371-9e64-fbfa54e26e35","Type":"ContainerStarted","Data":"a3eaed066c0c51fee61c54d9ef088f8f26ff4ba25aff9add2c316cfd73eb1bcf"} Oct 30 23:13:14.440292 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:14.440237 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-mpr8c" podStartSLOduration=2.55718824 podStartE2EDuration="5.440218227s" totalImagesPullingTime="2.883029987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:13:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:13:14.438366015 +0000 UTC m=+413.582196247" watchObservedRunningTime="2026-10-30 23:13:14.440218227 +0000 UTC m=+413.584048433" Oct 30 23:13:15.333934 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:15.333904 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-75gxr" event={"ID":"fb7db2c7-13a2-44b6-83f3-b5966cff4343","Type":"ContainerStarted","Data":"0ccdce472c74cfb038dcede7d2d95cf277febd6eac8b362b9faf4e3ad1ddf5c0"} Oct 30 23:13:15.335522 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:15.335493 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-qxr4b" event={"ID":"fb5cde51-c849-4371-9e64-fbfa54e26e35","Type":"ContainerStarted","Data":"2c8d051c984326c49585f72e8219d61d3bd15b091e5cfd1ccb4c2d50511a1ad9"} Oct 30 23:13:15.446861 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:15.446767 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-75gxr" podStartSLOduration=1.536353481 podStartE2EDuration="7.446755561s" totalImagesPullingTime="5.91040208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:13:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:13:15.445899176 +0000 UTC m=+414.589729384" watchObservedRunningTime="2026-10-30 23:13:15.446755561 +0000 UTC m=+414.590585768" Oct 30 23:13:15.561986 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:15.561937 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-qxr4b" podStartSLOduration=1.226361761 podStartE2EDuration="2.561921891s" totalImagesPullingTime="1.33556013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:13:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:13:15.560138016 +0000 UTC m=+414.703968235" watchObservedRunningTime="2026-10-30 23:13:15.561921891 +0000 UTC m=+414.705752095" Oct 30 23:13:15.600077 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:15.598248 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-mpr8c"] Oct 30 23:13:16.342352 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:16.342306 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-hj2mm" event={"ID":"52beb09a-acb7-4f24-a70b-195f1939ba32","Type":"ContainerStarted","Data":"f423a03bc511455dcf5feb184572f8fe480adb8d56afabde5eb24bec258b756c"} Oct 30 23:13:16.342836 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:16.342487 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-mpr8c" podUID="eea7057e-988a-456f-8b89-d909848d8bb7" containerName="authorino" containerID="cri-o://8d647df6ac3199154d1c75c81bd8383b2689082536e8e22cd598b6458cc48d94" gracePeriod=30 Oct 30 23:13:16.568574 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:16.568518 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-hj2mm" podStartSLOduration=1.68101269 podStartE2EDuration="3.568500889s" totalImagesPullingTime="1.887488199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:13:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:13:16.566475894 +0000 UTC m=+415.710306119" watchObservedRunningTime="2026-10-30 23:13:16.568500889 +0000 UTC m=+415.712331093" Oct 30 23:13:16.634320 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:16.634287 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-mpr8c" Oct 30 23:13:16.685525 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:16.685496 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eea7057e-988a-456f-8b89-d909848d8bb7-kube-api-access-mwppc\" (UniqueName: \"kubernetes.io/projected/eea7057e-988a-456f-8b89-d909848d8bb7-kube-api-access-mwppc\") pod \"eea7057e-988a-456f-8b89-d909848d8bb7\" (UID: \"eea7057e-988a-456f-8b89-d909848d8bb7\") " Oct 30 23:13:16.687759 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:16.687734 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea7057e-988a-456f-8b89-d909848d8bb7-kube-api-access-mwppc" pod "eea7057e-988a-456f-8b89-d909848d8bb7" (UID: "eea7057e-988a-456f-8b89-d909848d8bb7"). InnerVolumeSpecName "kube-api-access-mwppc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:13:16.786900 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:16.786869 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwppc\" (UniqueName: \"kubernetes.io/projected/eea7057e-988a-456f-8b89-d909848d8bb7-kube-api-access-mwppc\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:13:17.029256 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:17.029221 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:13:17.266955 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:17.266882 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:13:17.348797 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:17.348767 4952 generic.go:356] "Generic (PLEG): container finished" podID="eea7057e-988a-456f-8b89-d909848d8bb7" containerID="8d647df6ac3199154d1c75c81bd8383b2689082536e8e22cd598b6458cc48d94" exitCode=0 Oct 30 23:13:17.349250 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:17.348812 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-mpr8c" Oct 30 23:13:17.349250 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:17.348846 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-mpr8c" event={"ID":"eea7057e-988a-456f-8b89-d909848d8bb7","Type":"ContainerDied","Data":"8d647df6ac3199154d1c75c81bd8383b2689082536e8e22cd598b6458cc48d94"} Oct 30 23:13:17.349250 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:17.348889 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-mpr8c" event={"ID":"eea7057e-988a-456f-8b89-d909848d8bb7","Type":"ContainerDied","Data":"07bb1af5c4762d0aeef145e8c0302e901f9eadeceb58e0c9fa9bac05594a0015"} Oct 30 23:13:17.349250 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:17.348906 4952 scope.go:122] "RemoveContainer" containerID="8d647df6ac3199154d1c75c81bd8383b2689082536e8e22cd598b6458cc48d94" Oct 30 23:13:17.349250 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:17.349166 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-hj2mm" Oct 30 23:13:17.357523 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:17.357503 4952 scope.go:122] "RemoveContainer" containerID="8d647df6ac3199154d1c75c81bd8383b2689082536e8e22cd598b6458cc48d94" Oct 30 23:13:17.357802 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:13:17.357781 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d647df6ac3199154d1c75c81bd8383b2689082536e8e22cd598b6458cc48d94\": container with ID starting with 8d647df6ac3199154d1c75c81bd8383b2689082536e8e22cd598b6458cc48d94 not found: ID does not exist" containerID="8d647df6ac3199154d1c75c81bd8383b2689082536e8e22cd598b6458cc48d94" Oct 30 23:13:17.357895 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:17.357808 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d647df6ac3199154d1c75c81bd8383b2689082536e8e22cd598b6458cc48d94"} err="failed to get container status \"8d647df6ac3199154d1c75c81bd8383b2689082536e8e22cd598b6458cc48d94\": rpc error: code = NotFound desc = could not find container \"8d647df6ac3199154d1c75c81bd8383b2689082536e8e22cd598b6458cc48d94\": container with ID starting with 8d647df6ac3199154d1c75c81bd8383b2689082536e8e22cd598b6458cc48d94 not found: ID does not exist" Oct 30 23:13:17.511426 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:17.511394 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-mpr8c"] Oct 30 23:13:17.517345 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:17.517291 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-mpr8c"] Oct 30 23:13:17.691534 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:17.691492 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:13:17.919311 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:17.919280 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:13:18.969611 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:18.969579 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eea7057e-988a-456f-8b89-d909848d8bb7" path="/var/lib/kubelet/pods/eea7057e-988a-456f-8b89-d909848d8bb7/volumes" Oct 30 23:13:20.018717 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:20.018653 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:13:20.244608 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:20.244573 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:13:21.141149 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:13:21.141103 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:13:25.227608 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:25.227566 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:13:25.452231 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:25.452196 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:13:28.356132 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:28.356105 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-hj2mm" Oct 30 23:13:28.980566 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:28.980525 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:13:29.203515 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:29.203479 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:13:29.353202 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:29.353173 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-gl24g"] Oct 30 23:13:29.353410 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:29.353389 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-gl24g" podUID="9581e345-633d-4e29-beea-cc64366ef77f" containerName="manager" containerID="cri-o://65e648e6c2a2e8fe252ad0b386d7d29d2761a98eda8bd1d101ff74fe5be48a6d" gracePeriod=10 Oct 30 23:13:30.423702 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:30.423666 4952 generic.go:356] "Generic (PLEG): container finished" podID="9581e345-633d-4e29-beea-cc64366ef77f" containerID="65e648e6c2a2e8fe252ad0b386d7d29d2761a98eda8bd1d101ff74fe5be48a6d" exitCode=0 Oct 30 23:13:30.424144 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:30.423765 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-gl24g" event={"ID":"9581e345-633d-4e29-beea-cc64366ef77f","Type":"ContainerDied","Data":"65e648e6c2a2e8fe252ad0b386d7d29d2761a98eda8bd1d101ff74fe5be48a6d"} Oct 30 23:13:30.505865 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:30.505840 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-gl24g" Oct 30 23:13:30.601898 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:30.601835 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9581e345-633d-4e29-beea-cc64366ef77f-kube-api-access-8zlr7\" (UniqueName: \"kubernetes.io/projected/9581e345-633d-4e29-beea-cc64366ef77f-kube-api-access-8zlr7\") pod \"9581e345-633d-4e29-beea-cc64366ef77f\" (UID: \"9581e345-633d-4e29-beea-cc64366ef77f\") " Oct 30 23:13:30.601898 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:30.601890 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9581e345-633d-4e29-beea-cc64366ef77f-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9581e345-633d-4e29-beea-cc64366ef77f-extensions-socket-volume\") pod \"9581e345-633d-4e29-beea-cc64366ef77f\" (UID: \"9581e345-633d-4e29-beea-cc64366ef77f\") " Oct 30 23:13:30.602326 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:30.602304 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9581e345-633d-4e29-beea-cc64366ef77f-extensions-socket-volume" pod "9581e345-633d-4e29-beea-cc64366ef77f" (UID: "9581e345-633d-4e29-beea-cc64366ef77f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:13:30.604146 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:30.604121 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9581e345-633d-4e29-beea-cc64366ef77f-kube-api-access-8zlr7" pod "9581e345-633d-4e29-beea-cc64366ef77f" (UID: "9581e345-633d-4e29-beea-cc64366ef77f"). InnerVolumeSpecName "kube-api-access-8zlr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:13:30.702581 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:30.702544 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zlr7\" (UniqueName: \"kubernetes.io/projected/9581e345-633d-4e29-beea-cc64366ef77f-kube-api-access-8zlr7\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:13:30.702723 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:30.702584 4952 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9581e345-633d-4e29-beea-cc64366ef77f-extensions-socket-volume\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:13:31.430322 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:31.430294 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-gl24g" Oct 30 23:13:31.430322 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:31.430312 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-gl24g" event={"ID":"9581e345-633d-4e29-beea-cc64366ef77f","Type":"ContainerDied","Data":"aa80b6063e49d8c86de63f09cc8437062c910458d95a18f3ad61ed6f23fd0c30"} Oct 30 23:13:31.430784 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:31.430368 4952 scope.go:122] "RemoveContainer" containerID="65e648e6c2a2e8fe252ad0b386d7d29d2761a98eda8bd1d101ff74fe5be48a6d" Oct 30 23:13:31.643233 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:31.643206 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-gl24g"] Oct 30 23:13:31.692956 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:31.692894 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-gl24g"] Oct 30 23:13:32.968894 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:32.968862 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9581e345-633d-4e29-beea-cc64366ef77f" path="/var/lib/kubelet/pods/9581e345-633d-4e29-beea-cc64366ef77f/volumes" Oct 30 23:13:36.657655 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:36.657599 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:13:36.926101 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:36.926003 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:13:51.141635 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:13:51.141599 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:13:55.813281 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:55.813248 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:13:57.456921 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:57.456887 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:13:57.794717 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:57.794634 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:13:58.050249 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:58.050165 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-p4jb6"] Oct 30 23:13:58.050607 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:58.050590 4952 cpu_manager.go:414] "containerMap: removing container" podUID="eea7057e-988a-456f-8b89-d909848d8bb7" containerName="authorino" Oct 30 23:13:58.050700 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:58.050610 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eea7057e-988a-456f-8b89-d909848d8bb7" containerName="authorino" Oct 30 23:13:58.050700 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:58.050627 4952 cpu_manager.go:414] "containerMap: removing container" podUID="9581e345-633d-4e29-beea-cc64366ef77f" containerName="manager" Oct 30 23:13:58.050700 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:58.050633 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9581e345-633d-4e29-beea-cc64366ef77f" containerName="manager" Oct 30 23:13:58.050847 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:58.050777 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="eea7057e-988a-456f-8b89-d909848d8bb7" containerName="authorino" Oct 30 23:13:58.050847 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:58.050790 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="9581e345-633d-4e29-beea-cc64366ef77f" containerName="manager" Oct 30 23:13:58.065691 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:58.065663 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p4jb6" Oct 30 23:13:58.110601 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:58.110568 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-p4jb6"] Oct 30 23:13:58.125434 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:58.125408 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d6322b74-8f5a-46c5-98fc-0640e3a060a1-data\") pod \"postgres-7cdbf6dc44-p4jb6\" (UID: \"d6322b74-8f5a-46c5-98fc-0640e3a060a1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p4jb6" Oct 30 23:13:58.125575 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:58.125487 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc882\" (UniqueName: \"kubernetes.io/projected/d6322b74-8f5a-46c5-98fc-0640e3a060a1-kube-api-access-lc882\") pod \"postgres-7cdbf6dc44-p4jb6\" (UID: \"d6322b74-8f5a-46c5-98fc-0640e3a060a1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p4jb6" Oct 30 23:13:58.157683 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:58.157657 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 23:13:58.157837 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:58.157691 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-m84bg\"" Oct 30 23:13:58.168585 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:58.168562 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 23:13:58.168667 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:58.168602 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 23:13:58.225865 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:58.225838 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d6322b74-8f5a-46c5-98fc-0640e3a060a1-data\") pod \"postgres-7cdbf6dc44-p4jb6\" (UID: \"d6322b74-8f5a-46c5-98fc-0640e3a060a1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p4jb6" Oct 30 23:13:58.226028 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:58.225894 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc882\" (UniqueName: \"kubernetes.io/projected/d6322b74-8f5a-46c5-98fc-0640e3a060a1-kube-api-access-lc882\") pod \"postgres-7cdbf6dc44-p4jb6\" (UID: \"d6322b74-8f5a-46c5-98fc-0640e3a060a1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p4jb6" Oct 30 23:13:58.226297 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:58.226277 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d6322b74-8f5a-46c5-98fc-0640e3a060a1-data\") pod \"postgres-7cdbf6dc44-p4jb6\" (UID: \"d6322b74-8f5a-46c5-98fc-0640e3a060a1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p4jb6" Oct 30 23:13:58.241116 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:58.241089 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:13:58.282957 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:58.282929 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc882\" (UniqueName: \"kubernetes.io/projected/d6322b74-8f5a-46c5-98fc-0640e3a060a1-kube-api-access-lc882\") pod \"postgres-7cdbf6dc44-p4jb6\" (UID: \"d6322b74-8f5a-46c5-98fc-0640e3a060a1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p4jb6" Oct 30 23:13:58.381777 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:58.381749 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p4jb6" Oct 30 23:13:58.755956 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:58.755925 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-p4jb6"] Oct 30 23:13:58.757444 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:13:58.757417 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6322b74_8f5a_46c5_98fc_0640e3a060a1.slice/crio-d46d702d45e635617a5d84848ca3b001970dad99149af02e6a01e56ea313f94b WatchSource:0}: Error finding container d46d702d45e635617a5d84848ca3b001970dad99149af02e6a01e56ea313f94b: Status 404 returned error can't find the container with id d46d702d45e635617a5d84848ca3b001970dad99149af02e6a01e56ea313f94b Oct 30 23:13:59.579049 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:13:59.579006 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p4jb6" event={"ID":"d6322b74-8f5a-46c5-98fc-0640e3a060a1","Type":"ContainerStarted","Data":"d46d702d45e635617a5d84848ca3b001970dad99149af02e6a01e56ea313f94b"} Oct 30 23:14:06.621488 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:06.621455 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p4jb6" event={"ID":"d6322b74-8f5a-46c5-98fc-0640e3a060a1","Type":"ContainerStarted","Data":"af50e7fb3203f5048121f691d9d2e533624d84514a013c647163e0f640e129d0"} Oct 30 23:14:06.621859 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:06.621517 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p4jb6" Oct 30 23:14:06.736603 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:06.736548 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p4jb6" podStartSLOduration=1.839510497 podStartE2EDuration="8.736534454s" totalImagesPullingTime="6.897023957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:13:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:14:06.734220474 +0000 UTC m=+465.878050679" watchObservedRunningTime="2026-10-30 23:14:06.736534454 +0000 UTC m=+465.880364667" Oct 30 23:14:12.658445 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:12.658419 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p4jb6" Oct 30 23:14:13.078694 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:13.078571 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:14:19.905092 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:19.905062 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-ktfhs"] Oct 30 23:14:19.914618 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:19.914588 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:14:19.922813 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:19.922793 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-ktfhs" Oct 30 23:14:20.004386 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:20.004365 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 23:14:20.005796 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:20.005767 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4v8z\" (UniqueName: \"kubernetes.io/projected/f6a100d3-207e-46f2-a8cd-dae78e74d078-kube-api-access-g4v8z\") pod \"maas-controller-579cf7c6d-ktfhs\" (UID: \"f6a100d3-207e-46f2-a8cd-dae78e74d078\") " pod="opendatahub/maas-controller-579cf7c6d-ktfhs" Oct 30 23:14:20.005933 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:20.005809 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a100d3-207e-46f2-a8cd-dae78e74d078-metrics-certs\") pod \"maas-controller-579cf7c6d-ktfhs\" (UID: \"f6a100d3-207e-46f2-a8cd-dae78e74d078\") " pod="opendatahub/maas-controller-579cf7c6d-ktfhs" Oct 30 23:14:20.005933 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:20.005839 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f6a100d3-207e-46f2-a8cd-dae78e74d078-webhook-cert\") pod \"maas-controller-579cf7c6d-ktfhs\" (UID: \"f6a100d3-207e-46f2-a8cd-dae78e74d078\") " pod="opendatahub/maas-controller-579cf7c6d-ktfhs" Oct 30 23:14:20.025392 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:20.025370 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 23:14:20.025392 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:20.025369 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 23:14:20.025554 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:20.025411 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-txfhs\"" Oct 30 23:14:20.025554 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:20.025415 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 23:14:20.025671 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:20.025656 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 23:14:20.027785 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:20.027769 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-ktfhs"] Oct 30 23:14:20.106268 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:20.106242 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4v8z\" (UniqueName: \"kubernetes.io/projected/f6a100d3-207e-46f2-a8cd-dae78e74d078-kube-api-access-g4v8z\") pod \"maas-controller-579cf7c6d-ktfhs\" (UID: \"f6a100d3-207e-46f2-a8cd-dae78e74d078\") " pod="opendatahub/maas-controller-579cf7c6d-ktfhs" Oct 30 23:14:20.106413 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:20.106276 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a100d3-207e-46f2-a8cd-dae78e74d078-metrics-certs\") pod \"maas-controller-579cf7c6d-ktfhs\" (UID: \"f6a100d3-207e-46f2-a8cd-dae78e74d078\") " pod="opendatahub/maas-controller-579cf7c6d-ktfhs" Oct 30 23:14:20.106413 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:20.106307 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f6a100d3-207e-46f2-a8cd-dae78e74d078-webhook-cert\") pod \"maas-controller-579cf7c6d-ktfhs\" (UID: \"f6a100d3-207e-46f2-a8cd-dae78e74d078\") " pod="opendatahub/maas-controller-579cf7c6d-ktfhs" Oct 30 23:14:20.108844 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:20.108822 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f6a100d3-207e-46f2-a8cd-dae78e74d078-webhook-cert\") pod \"maas-controller-579cf7c6d-ktfhs\" (UID: \"f6a100d3-207e-46f2-a8cd-dae78e74d078\") " pod="opendatahub/maas-controller-579cf7c6d-ktfhs" Oct 30 23:14:20.108950 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:20.108825 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a100d3-207e-46f2-a8cd-dae78e74d078-metrics-certs\") pod \"maas-controller-579cf7c6d-ktfhs\" (UID: \"f6a100d3-207e-46f2-a8cd-dae78e74d078\") " pod="opendatahub/maas-controller-579cf7c6d-ktfhs" Oct 30 23:14:20.148587 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:20.148555 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:14:20.164502 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:20.164455 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4v8z\" (UniqueName: \"kubernetes.io/projected/f6a100d3-207e-46f2-a8cd-dae78e74d078-kube-api-access-g4v8z\") pod \"maas-controller-579cf7c6d-ktfhs\" (UID: \"f6a100d3-207e-46f2-a8cd-dae78e74d078\") " pod="opendatahub/maas-controller-579cf7c6d-ktfhs" Oct 30 23:14:20.234548 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:20.234525 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-ktfhs" Oct 30 23:14:20.318397 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:20.318368 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7bbf6fc889-b89vm"] Oct 30 23:14:20.321167 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:20.321113 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbf6fc889-b89vm" Oct 30 23:14:20.381001 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:20.380972 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bbf6fc889-b89vm"] Oct 30 23:14:20.386793 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:20.386761 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:14:20.409008 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:20.408981 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/da1faa77-ca82-428f-8d34-c602c4238182-webhook-cert\") pod \"maas-controller-7bbf6fc889-b89vm\" (UID: \"da1faa77-ca82-428f-8d34-c602c4238182\") " pod="opendatahub/maas-controller-7bbf6fc889-b89vm" Oct 30 23:14:20.409157 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:20.409016 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzcdp\" (UniqueName: \"kubernetes.io/projected/da1faa77-ca82-428f-8d34-c602c4238182-kube-api-access-wzcdp\") pod \"maas-controller-7bbf6fc889-b89vm\" (UID: \"da1faa77-ca82-428f-8d34-c602c4238182\") " pod="opendatahub/maas-controller-7bbf6fc889-b89vm" Oct 30 23:14:20.409157 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:20.409057 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da1faa77-ca82-428f-8d34-c602c4238182-metrics-certs\") pod \"maas-controller-7bbf6fc889-b89vm\" (UID: \"da1faa77-ca82-428f-8d34-c602c4238182\") " pod="opendatahub/maas-controller-7bbf6fc889-b89vm" Oct 30 23:14:20.510231 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:20.510172 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da1faa77-ca82-428f-8d34-c602c4238182-metrics-certs\") pod \"maas-controller-7bbf6fc889-b89vm\" (UID: \"da1faa77-ca82-428f-8d34-c602c4238182\") " pod="opendatahub/maas-controller-7bbf6fc889-b89vm" Oct 30 23:14:20.510368 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:20.510265 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/da1faa77-ca82-428f-8d34-c602c4238182-webhook-cert\") pod \"maas-controller-7bbf6fc889-b89vm\" (UID: \"da1faa77-ca82-428f-8d34-c602c4238182\") " pod="opendatahub/maas-controller-7bbf6fc889-b89vm" Oct 30 23:14:20.510368 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:20.510288 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzcdp\" (UniqueName: \"kubernetes.io/projected/da1faa77-ca82-428f-8d34-c602c4238182-kube-api-access-wzcdp\") pod \"maas-controller-7bbf6fc889-b89vm\" (UID: \"da1faa77-ca82-428f-8d34-c602c4238182\") " pod="opendatahub/maas-controller-7bbf6fc889-b89vm" Oct 30 23:14:20.512765 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:20.512743 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/da1faa77-ca82-428f-8d34-c602c4238182-webhook-cert\") pod \"maas-controller-7bbf6fc889-b89vm\" (UID: \"da1faa77-ca82-428f-8d34-c602c4238182\") " pod="opendatahub/maas-controller-7bbf6fc889-b89vm" Oct 30 23:14:20.512850 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:20.512766 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da1faa77-ca82-428f-8d34-c602c4238182-metrics-certs\") pod \"maas-controller-7bbf6fc889-b89vm\" (UID: \"da1faa77-ca82-428f-8d34-c602c4238182\") " pod="opendatahub/maas-controller-7bbf6fc889-b89vm" Oct 30 23:14:20.574744 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:20.574717 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzcdp\" (UniqueName: \"kubernetes.io/projected/da1faa77-ca82-428f-8d34-c602c4238182-kube-api-access-wzcdp\") pod \"maas-controller-7bbf6fc889-b89vm\" (UID: \"da1faa77-ca82-428f-8d34-c602c4238182\") " pod="opendatahub/maas-controller-7bbf6fc889-b89vm" Oct 30 23:14:20.630186 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:20.630163 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-ktfhs"] Oct 30 23:14:20.630896 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:14:20.630870 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a100d3_207e_46f2_a8cd_dae78e74d078.slice/crio-931a397840854eb2b76b2fafbefdc562ab1e884a0407f61de8104e8f1a82f3ca WatchSource:0}: Error finding container 931a397840854eb2b76b2fafbefdc562ab1e884a0407f61de8104e8f1a82f3ca: Status 404 returned error can't find the container with id 931a397840854eb2b76b2fafbefdc562ab1e884a0407f61de8104e8f1a82f3ca Oct 30 23:14:20.632736 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:20.632712 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbf6fc889-b89vm" Oct 30 23:14:20.698416 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:20.698386 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-ktfhs" event={"ID":"f6a100d3-207e-46f2-a8cd-dae78e74d078","Type":"ContainerStarted","Data":"931a397840854eb2b76b2fafbefdc562ab1e884a0407f61de8104e8f1a82f3ca"} Oct 30 23:14:21.004894 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:21.004866 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bbf6fc889-b89vm"] Oct 30 23:14:21.006354 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:14:21.006332 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda1faa77_ca82_428f_8d34_c602c4238182.slice/crio-061320028cce9f6c82a5fdfb3cd9ca71bb09cdd67c4442a53a410c484a60b434 WatchSource:0}: Error finding container 061320028cce9f6c82a5fdfb3cd9ca71bb09cdd67c4442a53a410c484a60b434: Status 404 returned error can't find the container with id 061320028cce9f6c82a5fdfb3cd9ca71bb09cdd67c4442a53a410c484a60b434 Oct 30 23:14:21.141061 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:14:21.141014 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:14:21.720802 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:21.720763 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbf6fc889-b89vm" event={"ID":"da1faa77-ca82-428f-8d34-c602c4238182","Type":"ContainerStarted","Data":"061320028cce9f6c82a5fdfb3cd9ca71bb09cdd67c4442a53a410c484a60b434"} Oct 30 23:14:24.009136 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:24.009100 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 23:14:24.647970 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:24.647924 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:14:24.740116 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:24.740068 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-ktfhs" event={"ID":"f6a100d3-207e-46f2-a8cd-dae78e74d078","Type":"ContainerStarted","Data":"de39e4df445126eaf8e5b4116fae6210ef69cfb0355249f8aa7c6133a0f6a279"} Oct 30 23:14:24.740280 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:24.740266 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-ktfhs" Oct 30 23:14:24.741632 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:24.741607 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbf6fc889-b89vm" event={"ID":"da1faa77-ca82-428f-8d34-c602c4238182","Type":"ContainerStarted","Data":"1592739455dc16ebf7716590b945c88a14f75b5696d8045c88ee97d145b0cef7"} Oct 30 23:14:24.741756 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:24.741647 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7bbf6fc889-b89vm" Oct 30 23:14:24.895871 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:24.895814 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-ktfhs" podStartSLOduration=2.572715526 podStartE2EDuration="5.895800627s" totalImagesPullingTime="3.323085101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:14:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:14:24.895352936 +0000 UTC m=+484.039183142" watchObservedRunningTime="2026-10-30 23:14:24.895800627 +0000 UTC m=+484.039630832" Oct 30 23:14:25.034160 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:25.034030 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7bbf6fc889-b89vm" podStartSLOduration=1.792459209 podStartE2EDuration="5.034017306s" totalImagesPullingTime="3.241558097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:14:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:14:25.032446358 +0000 UTC m=+484.176276565" watchObservedRunningTime="2026-10-30 23:14:25.034017306 +0000 UTC m=+484.177847511" Oct 30 23:14:25.546023 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:25.545979 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:14:31.775756 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:31.775666 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-69fdv"] Oct 30 23:14:31.779272 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:31.779248 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-69fdv" Oct 30 23:14:31.835712 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:31.835686 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 30 23:14:31.838147 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:31.838118 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-69fdv"] Oct 30 23:14:31.885277 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:31.885250 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-h6gwm\"" Oct 30 23:14:31.885443 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:31.885321 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 30 23:14:31.885443 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:31.885366 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 30 23:14:31.907859 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:31.907835 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5770149-5467-499e-8814-e52899bc3edc-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-69fdv\" (UID: \"d5770149-5467-499e-8814-e52899bc3edc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-69fdv" Oct 30 23:14:31.907974 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:31.907888 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d5770149-5467-499e-8814-e52899bc3edc-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-69fdv\" (UID: \"d5770149-5467-499e-8814-e52899bc3edc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-69fdv" Oct 30 23:14:31.908076 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:31.907969 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6dlv\" (UniqueName: \"kubernetes.io/projected/d5770149-5467-499e-8814-e52899bc3edc-kube-api-access-z6dlv\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-69fdv\" (UID: \"d5770149-5467-499e-8814-e52899bc3edc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-69fdv" Oct 30 23:14:31.908076 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:31.908014 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/d5770149-5467-499e-8814-e52899bc3edc-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-69fdv\" (UID: \"d5770149-5467-499e-8814-e52899bc3edc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-69fdv" Oct 30 23:14:32.008764 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:32.008730 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d5770149-5467-499e-8814-e52899bc3edc-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-69fdv\" (UID: \"d5770149-5467-499e-8814-e52899bc3edc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-69fdv" Oct 30 23:14:32.008899 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:32.008787 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6dlv\" (UniqueName: \"kubernetes.io/projected/d5770149-5467-499e-8814-e52899bc3edc-kube-api-access-z6dlv\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-69fdv\" (UID: \"d5770149-5467-499e-8814-e52899bc3edc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-69fdv" Oct 30 23:14:32.008899 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:32.008805 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/d5770149-5467-499e-8814-e52899bc3edc-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-69fdv\" (UID: \"d5770149-5467-499e-8814-e52899bc3edc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-69fdv" Oct 30 23:14:32.008899 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:32.008839 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5770149-5467-499e-8814-e52899bc3edc-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-69fdv\" (UID: \"d5770149-5467-499e-8814-e52899bc3edc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-69fdv" Oct 30 23:14:32.008899 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:14:32.008855 4952 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Oct 30 23:14:32.009105 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:14:32.008920 4952 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Oct 30 23:14:32.009105 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:14:32.008925 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5770149-5467-499e-8814-e52899bc3edc-tls-cert podName:d5770149-5467-499e-8814-e52899bc3edc nodeName:}" failed. No retries permitted until 2026-10-30 23:14:32.508903062 +0000 UTC m=+491.652733245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/d5770149-5467-499e-8814-e52899bc3edc-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-69fdv" (UID: "d5770149-5467-499e-8814-e52899bc3edc") : secret "usage-logs-tenancy-proxy-tls" not found Oct 30 23:14:32.009105 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:14:32.008978 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5770149-5467-499e-8814-e52899bc3edc-proxy-source podName:d5770149-5467-499e-8814-e52899bc3edc nodeName:}" failed. No retries permitted until 2026-10-30 23:14:32.508962229 +0000 UTC m=+491.652792429 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/d5770149-5467-499e-8814-e52899bc3edc-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-69fdv" (UID: "d5770149-5467-499e-8814-e52899bc3edc") : configmap "usage-tenancy-proxy-source" not found Oct 30 23:14:32.009468 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:32.009449 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5770149-5467-499e-8814-e52899bc3edc-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-69fdv\" (UID: \"d5770149-5467-499e-8814-e52899bc3edc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-69fdv" Oct 30 23:14:32.022324 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:32.022300 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:14:32.066493 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:32.066442 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6dlv\" (UniqueName: \"kubernetes.io/projected/d5770149-5467-499e-8814-e52899bc3edc-kube-api-access-z6dlv\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-69fdv\" (UID: \"d5770149-5467-499e-8814-e52899bc3edc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-69fdv" Oct 30 23:14:32.318939 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:32.318862 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:14:32.514541 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:32.514508 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d5770149-5467-499e-8814-e52899bc3edc-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-69fdv\" (UID: \"d5770149-5467-499e-8814-e52899bc3edc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-69fdv" Oct 30 23:14:32.514705 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:32.514574 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/d5770149-5467-499e-8814-e52899bc3edc-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-69fdv\" (UID: \"d5770149-5467-499e-8814-e52899bc3edc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-69fdv" Oct 30 23:14:32.515391 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:32.515373 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/d5770149-5467-499e-8814-e52899bc3edc-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-69fdv\" (UID: \"d5770149-5467-499e-8814-e52899bc3edc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-69fdv" Oct 30 23:14:32.517207 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:32.517188 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d5770149-5467-499e-8814-e52899bc3edc-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-69fdv\" (UID: \"d5770149-5467-499e-8814-e52899bc3edc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-69fdv" Oct 30 23:14:32.692186 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:32.692159 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-69fdv" Oct 30 23:14:33.072851 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:33.072824 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-69fdv"] Oct 30 23:14:33.074370 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:14:33.074343 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5770149_5467_499e_8814_e52899bc3edc.slice/crio-ce9580f365c1d113ba32f40eab8f9ac87381cb5215b051294e81e5b29764a41a WatchSource:0}: Error finding container ce9580f365c1d113ba32f40eab8f9ac87381cb5215b051294e81e5b29764a41a: Status 404 returned error can't find the container with id ce9580f365c1d113ba32f40eab8f9ac87381cb5215b051294e81e5b29764a41a Oct 30 23:14:33.789535 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:33.789504 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-69fdv" event={"ID":"d5770149-5467-499e-8814-e52899bc3edc","Type":"ContainerStarted","Data":"ce9580f365c1d113ba32f40eab8f9ac87381cb5215b051294e81e5b29764a41a"} Oct 30 23:14:33.900091 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:33.900052 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:14:34.186288 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:34.186182 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:14:34.503593 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:34.503482 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:14:34.931710 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:34.931676 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-qc829"] Oct 30 23:14:34.936530 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:34.936513 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qc829" Oct 30 23:14:34.961387 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:34.961353 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:14:34.995415 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:34.995391 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 23:14:34.997085 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:34.997067 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-vmbvh\"" Oct 30 23:14:35.037693 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:35.037668 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5rf5\" (UniqueName: \"kubernetes.io/projected/eb357cbd-85f5-4057-b7bb-852fb8465862-kube-api-access-z5rf5\") pod \"llmisvc-controller-manager-869d5dcc5d-qc829\" (UID: \"eb357cbd-85f5-4057-b7bb-852fb8465862\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qc829" Oct 30 23:14:35.037817 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:35.037732 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb357cbd-85f5-4057-b7bb-852fb8465862-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-qc829\" (UID: \"eb357cbd-85f5-4057-b7bb-852fb8465862\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qc829" Oct 30 23:14:35.047783 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:35.047763 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-qc829"] Oct 30 23:14:35.049756 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:35.049737 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-bxg8r"] Oct 30 23:14:35.053379 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:35.053360 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-bxg8r" Oct 30 23:14:35.113364 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:35.113341 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 23:14:35.121271 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:35.121250 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-ncmcx\"" Oct 30 23:14:35.126697 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:35.126670 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-bxg8r"] Oct 30 23:14:35.139210 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:35.139189 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb357cbd-85f5-4057-b7bb-852fb8465862-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-qc829\" (UID: \"eb357cbd-85f5-4057-b7bb-852fb8465862\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qc829" Oct 30 23:14:35.139351 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:35.139232 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/990ec266-ac95-4e83-a20e-503e8b5ee15e-tls-certs\") pod \"model-serving-api-75bd89fdfc-bxg8r\" (UID: \"990ec266-ac95-4e83-a20e-503e8b5ee15e\") " pod="opendatahub/model-serving-api-75bd89fdfc-bxg8r" Oct 30 23:14:35.139351 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:35.139309 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5rf5\" (UniqueName: \"kubernetes.io/projected/eb357cbd-85f5-4057-b7bb-852fb8465862-kube-api-access-z5rf5\") pod \"llmisvc-controller-manager-869d5dcc5d-qc829\" (UID: \"eb357cbd-85f5-4057-b7bb-852fb8465862\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qc829" Oct 30 23:14:35.139351 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:35.139335 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9jhk\" (UniqueName: \"kubernetes.io/projected/990ec266-ac95-4e83-a20e-503e8b5ee15e-kube-api-access-f9jhk\") pod \"model-serving-api-75bd89fdfc-bxg8r\" (UID: \"990ec266-ac95-4e83-a20e-503e8b5ee15e\") " pod="opendatahub/model-serving-api-75bd89fdfc-bxg8r" Oct 30 23:14:35.141797 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:35.141775 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb357cbd-85f5-4057-b7bb-852fb8465862-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-qc829\" (UID: \"eb357cbd-85f5-4057-b7bb-852fb8465862\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qc829" Oct 30 23:14:35.202075 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:35.201989 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5rf5\" (UniqueName: \"kubernetes.io/projected/eb357cbd-85f5-4057-b7bb-852fb8465862-kube-api-access-z5rf5\") pod \"llmisvc-controller-manager-869d5dcc5d-qc829\" (UID: \"eb357cbd-85f5-4057-b7bb-852fb8465862\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qc829" Oct 30 23:14:35.240341 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:35.240315 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9jhk\" (UniqueName: \"kubernetes.io/projected/990ec266-ac95-4e83-a20e-503e8b5ee15e-kube-api-access-f9jhk\") pod \"model-serving-api-75bd89fdfc-bxg8r\" (UID: \"990ec266-ac95-4e83-a20e-503e8b5ee15e\") " pod="opendatahub/model-serving-api-75bd89fdfc-bxg8r" Oct 30 23:14:35.240446 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:35.240382 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/990ec266-ac95-4e83-a20e-503e8b5ee15e-tls-certs\") pod \"model-serving-api-75bd89fdfc-bxg8r\" (UID: \"990ec266-ac95-4e83-a20e-503e8b5ee15e\") " pod="opendatahub/model-serving-api-75bd89fdfc-bxg8r" Oct 30 23:14:35.242722 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:35.242698 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/990ec266-ac95-4e83-a20e-503e8b5ee15e-tls-certs\") pod \"model-serving-api-75bd89fdfc-bxg8r\" (UID: \"990ec266-ac95-4e83-a20e-503e8b5ee15e\") " pod="opendatahub/model-serving-api-75bd89fdfc-bxg8r" Oct 30 23:14:35.249601 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:35.249583 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qc829" Oct 30 23:14:35.285824 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:35.285794 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:14:35.305000 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:35.304956 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9jhk\" (UniqueName: \"kubernetes.io/projected/990ec266-ac95-4e83-a20e-503e8b5ee15e-kube-api-access-f9jhk\") pod \"model-serving-api-75bd89fdfc-bxg8r\" (UID: \"990ec266-ac95-4e83-a20e-503e8b5ee15e\") " pod="opendatahub/model-serving-api-75bd89fdfc-bxg8r" Oct 30 23:14:35.366915 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:35.366885 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-bxg8r" Oct 30 23:14:35.630064 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:35.630019 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-qc829"] Oct 30 23:14:35.630827 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:14:35.630803 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podeb357cbd_85f5_4057_b7bb_852fb8465862.slice/crio-9397ab52de0a4eb643f6c41f84286295685ab71f58fbd46c50141c8c0837b79e WatchSource:0}: Error finding container 9397ab52de0a4eb643f6c41f84286295685ab71f58fbd46c50141c8c0837b79e: Status 404 returned error can't find the container with id 9397ab52de0a4eb643f6c41f84286295685ab71f58fbd46c50141c8c0837b79e Oct 30 23:14:35.749134 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:35.749098 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-ktfhs" Oct 30 23:14:35.749292 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:35.749222 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7bbf6fc889-b89vm" Oct 30 23:14:35.781696 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:35.781671 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-bxg8r"] Oct 30 23:14:35.784076 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:14:35.784017 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod990ec266_ac95_4e83_a20e_503e8b5ee15e.slice/crio-2036bafa479a5881b48671a1e75e3aadb4b002f49d16d0af3b1236f6c6180f43 WatchSource:0}: Error finding container 2036bafa479a5881b48671a1e75e3aadb4b002f49d16d0af3b1236f6c6180f43: Status 404 returned error can't find the container with id 2036bafa479a5881b48671a1e75e3aadb4b002f49d16d0af3b1236f6c6180f43 Oct 30 23:14:35.801095 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:35.801065 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qc829" event={"ID":"eb357cbd-85f5-4057-b7bb-852fb8465862","Type":"ContainerStarted","Data":"9397ab52de0a4eb643f6c41f84286295685ab71f58fbd46c50141c8c0837b79e"} Oct 30 23:14:35.802298 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:35.802270 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-bxg8r" event={"ID":"990ec266-ac95-4e83-a20e-503e8b5ee15e","Type":"ContainerStarted","Data":"2036bafa479a5881b48671a1e75e3aadb4b002f49d16d0af3b1236f6c6180f43"} Oct 30 23:14:36.236409 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:36.235382 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-ktfhs"] Oct 30 23:14:36.236409 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:36.235664 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-ktfhs" podUID="f6a100d3-207e-46f2-a8cd-dae78e74d078" containerName="manager" containerID="cri-o://de39e4df445126eaf8e5b4116fae6210ef69cfb0355249f8aa7c6133a0f6a279" gracePeriod=30 Oct 30 23:14:36.493772 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:36.493698 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:14:36.546946 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:36.546635 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-ktfhs" Oct 30 23:14:36.660495 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:36.660461 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6a100d3-207e-46f2-a8cd-dae78e74d078-webhook-cert\" (UniqueName: \"kubernetes.io/secret/f6a100d3-207e-46f2-a8cd-dae78e74d078-webhook-cert\") pod \"f6a100d3-207e-46f2-a8cd-dae78e74d078\" (UID: \"f6a100d3-207e-46f2-a8cd-dae78e74d078\") " Oct 30 23:14:36.660670 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:36.660578 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6a100d3-207e-46f2-a8cd-dae78e74d078-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a100d3-207e-46f2-a8cd-dae78e74d078-metrics-certs\") pod \"f6a100d3-207e-46f2-a8cd-dae78e74d078\" (UID: \"f6a100d3-207e-46f2-a8cd-dae78e74d078\") " Oct 30 23:14:36.660670 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:36.660618 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6a100d3-207e-46f2-a8cd-dae78e74d078-kube-api-access-g4v8z\" (UniqueName: \"kubernetes.io/projected/f6a100d3-207e-46f2-a8cd-dae78e74d078-kube-api-access-g4v8z\") pod \"f6a100d3-207e-46f2-a8cd-dae78e74d078\" (UID: \"f6a100d3-207e-46f2-a8cd-dae78e74d078\") " Oct 30 23:14:36.672300 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:36.672220 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a100d3-207e-46f2-a8cd-dae78e74d078-webhook-cert" pod "f6a100d3-207e-46f2-a8cd-dae78e74d078" (UID: "f6a100d3-207e-46f2-a8cd-dae78e74d078"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:14:36.674209 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:36.674176 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a100d3-207e-46f2-a8cd-dae78e74d078-metrics-certs" pod "f6a100d3-207e-46f2-a8cd-dae78e74d078" (UID: "f6a100d3-207e-46f2-a8cd-dae78e74d078"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:14:36.679526 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:36.678983 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a100d3-207e-46f2-a8cd-dae78e74d078-kube-api-access-g4v8z" pod "f6a100d3-207e-46f2-a8cd-dae78e74d078" (UID: "f6a100d3-207e-46f2-a8cd-dae78e74d078"). InnerVolumeSpecName "kube-api-access-g4v8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:14:36.761508 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:36.761427 4952 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f6a100d3-207e-46f2-a8cd-dae78e74d078-webhook-cert\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:14:36.761508 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:36.761462 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a100d3-207e-46f2-a8cd-dae78e74d078-metrics-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:14:36.761508 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:36.761475 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4v8z\" (UniqueName: \"kubernetes.io/projected/f6a100d3-207e-46f2-a8cd-dae78e74d078-kube-api-access-g4v8z\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:14:36.816550 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:36.816518 4952 generic.go:356] "Generic (PLEG): container finished" podID="f6a100d3-207e-46f2-a8cd-dae78e74d078" containerID="de39e4df445126eaf8e5b4116fae6210ef69cfb0355249f8aa7c6133a0f6a279" exitCode=0 Oct 30 23:14:36.816762 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:36.816736 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-ktfhs" event={"ID":"f6a100d3-207e-46f2-a8cd-dae78e74d078","Type":"ContainerDied","Data":"de39e4df445126eaf8e5b4116fae6210ef69cfb0355249f8aa7c6133a0f6a279"} Oct 30 23:14:36.816847 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:36.816830 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-ktfhs" event={"ID":"f6a100d3-207e-46f2-a8cd-dae78e74d078","Type":"ContainerDied","Data":"931a397840854eb2b76b2fafbefdc562ab1e884a0407f61de8104e8f1a82f3ca"} Oct 30 23:14:36.816910 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:36.816863 4952 scope.go:122] "RemoveContainer" containerID="de39e4df445126eaf8e5b4116fae6210ef69cfb0355249f8aa7c6133a0f6a279" Oct 30 23:14:36.817083 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:36.817064 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-ktfhs" Oct 30 23:14:36.844449 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:36.844412 4952 scope.go:122] "RemoveContainer" containerID="de39e4df445126eaf8e5b4116fae6210ef69cfb0355249f8aa7c6133a0f6a279" Oct 30 23:14:36.845779 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:14:36.845747 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de39e4df445126eaf8e5b4116fae6210ef69cfb0355249f8aa7c6133a0f6a279\": container with ID starting with de39e4df445126eaf8e5b4116fae6210ef69cfb0355249f8aa7c6133a0f6a279 not found: ID does not exist" containerID="de39e4df445126eaf8e5b4116fae6210ef69cfb0355249f8aa7c6133a0f6a279" Oct 30 23:14:36.845894 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:36.845790 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de39e4df445126eaf8e5b4116fae6210ef69cfb0355249f8aa7c6133a0f6a279"} err="failed to get container status \"de39e4df445126eaf8e5b4116fae6210ef69cfb0355249f8aa7c6133a0f6a279\": rpc error: code = NotFound desc = could not find container \"de39e4df445126eaf8e5b4116fae6210ef69cfb0355249f8aa7c6133a0f6a279\": container with ID starting with de39e4df445126eaf8e5b4116fae6210ef69cfb0355249f8aa7c6133a0f6a279 not found: ID does not exist" Oct 30 23:14:36.988007 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:36.987958 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-ktfhs"] Oct 30 23:14:36.991339 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:36.991313 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-ktfhs"] Oct 30 23:14:38.972726 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:38.972352 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6a100d3-207e-46f2-a8cd-dae78e74d078" path="/var/lib/kubelet/pods/f6a100d3-207e-46f2-a8cd-dae78e74d078/volumes" Oct 30 23:14:39.851576 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:39.851541 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-bxg8r" event={"ID":"990ec266-ac95-4e83-a20e-503e8b5ee15e","Type":"ContainerStarted","Data":"00a186fe9f20ded4664077c21ca9876ec00a445bbaf8dba716de716fffc96ec2"} Oct 30 23:14:39.851756 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:39.851641 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-bxg8r" Oct 30 23:14:39.970937 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:39.970887 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-bxg8r" podStartSLOduration=2.404278443 podStartE2EDuration="5.970874604s" totalImagesPullingTime="3.566596161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:14:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:14:39.96931157 +0000 UTC m=+499.113141806" watchObservedRunningTime="2026-10-30 23:14:39.970874604 +0000 UTC m=+499.114704810" Oct 30 23:14:40.861710 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:40.861672 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qc829" event={"ID":"eb357cbd-85f5-4057-b7bb-852fb8465862","Type":"ContainerStarted","Data":"aa69bb0b21819fb80129046f4dea82769b290051b4ce21a70f75879fca48c73b"} Oct 30 23:14:40.862215 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:40.861756 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qc829" Oct 30 23:14:40.978664 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:40.978620 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qc829" podStartSLOduration=2.063074841 podStartE2EDuration="6.978606094s" totalImagesPullingTime="4.915531253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:14:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:14:40.976556493 +0000 UTC m=+500.120386712" watchObservedRunningTime="2026-10-30 23:14:40.978606094 +0000 UTC m=+500.122436299" Oct 30 23:14:41.799064 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:41.799007 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:14:42.034221 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:42.034183 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:14:42.325923 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:42.325891 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:14:42.560520 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:42.560486 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:14:42.803853 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:42.803815 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:14:43.056150 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:43.055976 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:14:46.902104 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:46.902011 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-69fdv" event={"ID":"d5770149-5467-499e-8814-e52899bc3edc","Type":"ContainerStarted","Data":"d1843d6c6395b2649d7a97d2c6cb4ba938869c867534e5214aa18036d157d733"} Oct 30 23:14:46.902478 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:46.902110 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-69fdv" Oct 30 23:14:47.022341 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:47.022293 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-69fdv" podStartSLOduration=2.491603844 podStartE2EDuration="16.022279932s" totalImagesPullingTime="13.530676088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:14:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:14:47.021260267 +0000 UTC m=+506.165090471" watchObservedRunningTime="2026-10-30 23:14:47.022279932 +0000 UTC m=+506.166110137" Oct 30 23:14:47.642026 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:47.641993 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:14:50.865597 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:50.865572 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-bxg8r" Oct 30 23:14:51.141563 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:14:51.141483 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:14:57.915587 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:57.915552 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-69fdv" Oct 30 23:14:58.147837 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:58.147807 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:14:58.450026 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:14:58.449991 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:15:00.365681 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:00.365649 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:15:00.653213 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:00.653127 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:15:00.929733 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:00.929661 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:15:06.825800 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:06.825757 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:15:11.897466 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:11.897408 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qc829" Oct 30 23:15:12.245700 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:12.245619 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:15:12.418224 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:12.418191 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9s6ft"] Oct 30 23:15:12.425829 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:12.425789 4952 cpu_manager.go:414] "containerMap: removing container" podUID="f6a100d3-207e-46f2-a8cd-dae78e74d078" containerName="manager" Oct 30 23:15:12.425981 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:12.425927 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6a100d3-207e-46f2-a8cd-dae78e74d078" containerName="manager" Oct 30 23:15:12.426133 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:12.426118 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6a100d3-207e-46f2-a8cd-dae78e74d078" containerName="manager" Oct 30 23:15:12.432107 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:12.432086 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-9s6ft" Oct 30 23:15:12.497954 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:12.497857 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9s6ft"] Oct 30 23:15:12.511973 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:12.511945 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26zcf\" (UniqueName: \"kubernetes.io/projected/b8d8cdcb-8d67-425b-aa72-c9a4aef16479-kube-api-access-26zcf\") pod \"authorino-6f6944bcdd-9s6ft\" (UID: \"b8d8cdcb-8d67-425b-aa72-c9a4aef16479\") " pod="kuadrant-system/authorino-6f6944bcdd-9s6ft" Oct 30 23:15:12.519781 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:12.519751 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:15:12.612617 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:12.612579 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26zcf\" (UniqueName: \"kubernetes.io/projected/b8d8cdcb-8d67-425b-aa72-c9a4aef16479-kube-api-access-26zcf\") pod \"authorino-6f6944bcdd-9s6ft\" (UID: \"b8d8cdcb-8d67-425b-aa72-c9a4aef16479\") " pod="kuadrant-system/authorino-6f6944bcdd-9s6ft" Oct 30 23:15:12.695654 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:12.695617 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26zcf\" (UniqueName: \"kubernetes.io/projected/b8d8cdcb-8d67-425b-aa72-c9a4aef16479-kube-api-access-26zcf\") pod \"authorino-6f6944bcdd-9s6ft\" (UID: \"b8d8cdcb-8d67-425b-aa72-c9a4aef16479\") " pod="kuadrant-system/authorino-6f6944bcdd-9s6ft" Oct 30 23:15:12.745392 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:12.745355 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-9s6ft" Oct 30 23:15:12.797700 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:12.797667 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9s6ft"] Oct 30 23:15:12.826195 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:12.826165 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6948467c5c-czqzk"] Oct 30 23:15:12.833584 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:12.832345 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6948467c5c-czqzk" Oct 30 23:15:12.899023 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:12.898990 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6948467c5c-czqzk"] Oct 30 23:15:12.914957 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:12.914925 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzgcf\" (UniqueName: \"kubernetes.io/projected/62795711-cd58-437f-b82c-25dd7f019d04-kube-api-access-kzgcf\") pod \"authorino-6948467c5c-czqzk\" (UID: \"62795711-cd58-437f-b82c-25dd7f019d04\") " pod="kuadrant-system/authorino-6948467c5c-czqzk" Oct 30 23:15:13.016327 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:13.016237 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzgcf\" (UniqueName: \"kubernetes.io/projected/62795711-cd58-437f-b82c-25dd7f019d04-kube-api-access-kzgcf\") pod \"authorino-6948467c5c-czqzk\" (UID: \"62795711-cd58-437f-b82c-25dd7f019d04\") " pod="kuadrant-system/authorino-6948467c5c-czqzk" Oct 30 23:15:13.093162 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:13.093115 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzgcf\" (UniqueName: \"kubernetes.io/projected/62795711-cd58-437f-b82c-25dd7f019d04-kube-api-access-kzgcf\") pod \"authorino-6948467c5c-czqzk\" (UID: \"62795711-cd58-437f-b82c-25dd7f019d04\") " pod="kuadrant-system/authorino-6948467c5c-czqzk" Oct 30 23:15:13.155694 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:13.155661 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9s6ft"] Oct 30 23:15:13.156604 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:15:13.156568 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8d8cdcb_8d67_425b_aa72_c9a4aef16479.slice/crio-aa776ac08e984934e7187841615a60644d31b64c576d9365d2907347130f55d3 WatchSource:0}: Error finding container aa776ac08e984934e7187841615a60644d31b64c576d9365d2907347130f55d3: Status 404 returned error can't find the container with id aa776ac08e984934e7187841615a60644d31b64c576d9365d2907347130f55d3 Oct 30 23:15:13.159795 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:13.159772 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6948467c5c-czqzk" Oct 30 23:15:13.278408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:13.278339 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6948467c5c-czqzk"] Oct 30 23:15:13.310373 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:13.310343 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-595664bccf-bmvjk"] Oct 30 23:15:13.351999 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:13.351964 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-595664bccf-bmvjk" Oct 30 23:15:13.373412 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:13.373384 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-595664bccf-bmvjk"] Oct 30 23:15:13.405105 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:13.405078 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 23:15:13.421368 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:13.421338 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcrzm\" (UniqueName: \"kubernetes.io/projected/7a0b5138-0233-4c56-b97d-6a68d2d153f2-kube-api-access-mcrzm\") pod \"authorino-595664bccf-bmvjk\" (UID: \"7a0b5138-0233-4c56-b97d-6a68d2d153f2\") " pod="kuadrant-system/authorino-595664bccf-bmvjk" Oct 30 23:15:13.421496 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:13.421409 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7a0b5138-0233-4c56-b97d-6a68d2d153f2-tls-cert\") pod \"authorino-595664bccf-bmvjk\" (UID: \"7a0b5138-0233-4c56-b97d-6a68d2d153f2\") " pod="kuadrant-system/authorino-595664bccf-bmvjk" Oct 30 23:15:13.444467 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:13.444439 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:15:13.522689 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:13.522650 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcrzm\" (UniqueName: \"kubernetes.io/projected/7a0b5138-0233-4c56-b97d-6a68d2d153f2-kube-api-access-mcrzm\") pod \"authorino-595664bccf-bmvjk\" (UID: \"7a0b5138-0233-4c56-b97d-6a68d2d153f2\") " pod="kuadrant-system/authorino-595664bccf-bmvjk" Oct 30 23:15:13.522835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:13.522708 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7a0b5138-0233-4c56-b97d-6a68d2d153f2-tls-cert\") pod \"authorino-595664bccf-bmvjk\" (UID: \"7a0b5138-0233-4c56-b97d-6a68d2d153f2\") " pod="kuadrant-system/authorino-595664bccf-bmvjk" Oct 30 23:15:13.525376 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:13.525350 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7a0b5138-0233-4c56-b97d-6a68d2d153f2-tls-cert\") pod \"authorino-595664bccf-bmvjk\" (UID: \"7a0b5138-0233-4c56-b97d-6a68d2d153f2\") " pod="kuadrant-system/authorino-595664bccf-bmvjk" Oct 30 23:15:13.563325 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:13.563277 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6948467c5c-czqzk"] Oct 30 23:15:13.565099 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:15:13.565071 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62795711_cd58_437f_b82c_25dd7f019d04.slice/crio-5db53afcd84a08deac22e8ccce768ffbecc3d35b07f13d7ccee1e97de047485d WatchSource:0}: Error finding container 5db53afcd84a08deac22e8ccce768ffbecc3d35b07f13d7ccee1e97de047485d: Status 404 returned error can't find the container with id 5db53afcd84a08deac22e8ccce768ffbecc3d35b07f13d7ccee1e97de047485d Oct 30 23:15:13.582390 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:13.582362 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcrzm\" (UniqueName: \"kubernetes.io/projected/7a0b5138-0233-4c56-b97d-6a68d2d153f2-kube-api-access-mcrzm\") pod \"authorino-595664bccf-bmvjk\" (UID: \"7a0b5138-0233-4c56-b97d-6a68d2d153f2\") " pod="kuadrant-system/authorino-595664bccf-bmvjk" Oct 30 23:15:13.677299 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:13.677261 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-595664bccf-bmvjk" Oct 30 23:15:14.048097 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:14.048059 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6948467c5c-czqzk" event={"ID":"62795711-cd58-437f-b82c-25dd7f019d04","Type":"ContainerStarted","Data":"5db53afcd84a08deac22e8ccce768ffbecc3d35b07f13d7ccee1e97de047485d"} Oct 30 23:15:14.049493 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:14.049468 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-9s6ft" event={"ID":"b8d8cdcb-8d67-425b-aa72-c9a4aef16479","Type":"ContainerStarted","Data":"aa776ac08e984934e7187841615a60644d31b64c576d9365d2907347130f55d3"} Oct 30 23:15:14.077255 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:14.077222 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-595664bccf-bmvjk"] Oct 30 23:15:14.080341 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:15:14.080313 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a0b5138_0233_4c56_b97d_6a68d2d153f2.slice/crio-9295c4ab468dfe2d7eee005442431a7b7a54b73864fe2a965a4500da1fb703a0 WatchSource:0}: Error finding container 9295c4ab468dfe2d7eee005442431a7b7a54b73864fe2a965a4500da1fb703a0: Status 404 returned error can't find the container with id 9295c4ab468dfe2d7eee005442431a7b7a54b73864fe2a965a4500da1fb703a0 Oct 30 23:15:15.056291 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:15.056254 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-595664bccf-bmvjk" event={"ID":"7a0b5138-0233-4c56-b97d-6a68d2d153f2","Type":"ContainerStarted","Data":"9295c4ab468dfe2d7eee005442431a7b7a54b73864fe2a965a4500da1fb703a0"} Oct 30 23:15:16.067298 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:16.067254 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-9s6ft" event={"ID":"b8d8cdcb-8d67-425b-aa72-c9a4aef16479","Type":"ContainerStarted","Data":"b36878723293d90414c50d45ca4f003c005d4c737637c19ceb2519e91a97332f"} Oct 30 23:15:16.068587 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:16.068015 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-9s6ft" podUID="b8d8cdcb-8d67-425b-aa72-c9a4aef16479" containerName="authorino" containerID="cri-o://b36878723293d90414c50d45ca4f003c005d4c737637c19ceb2519e91a97332f" gracePeriod=30 Oct 30 23:15:16.071200 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:16.071166 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-595664bccf-bmvjk" event={"ID":"7a0b5138-0233-4c56-b97d-6a68d2d153f2","Type":"ContainerStarted","Data":"26309694a2fab6813db6865adfa4f1ecf943f0a3f5970825e165b5bc628a1063"} Oct 30 23:15:16.074468 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:16.074434 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6948467c5c-czqzk" event={"ID":"62795711-cd58-437f-b82c-25dd7f019d04","Type":"ContainerStarted","Data":"5de0030c898031d37f60883a34e6769b9b55c4f000c6ae9d6aea66090667ada7"} Oct 30 23:15:16.074653 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:16.074598 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6948467c5c-czqzk" podUID="62795711-cd58-437f-b82c-25dd7f019d04" containerName="authorino" containerID="cri-o://5de0030c898031d37f60883a34e6769b9b55c4f000c6ae9d6aea66090667ada7" gracePeriod=30 Oct 30 23:15:16.192063 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:16.183706 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-9s6ft" podStartSLOduration=2.257980474 podStartE2EDuration="4.18368821s" totalImagesPullingTime="1.925707736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:15:16.18317543 +0000 UTC m=+535.327005637" watchObservedRunningTime="2026-10-30 23:15:16.18368821 +0000 UTC m=+535.327518416" Oct 30 23:15:16.309806 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:16.309740 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6948467c5c-czqzk" podStartSLOduration=2.703201001 podStartE2EDuration="4.309719622s" totalImagesPullingTime="1.606518621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:15:16.306537189 +0000 UTC m=+535.450367394" watchObservedRunningTime="2026-10-30 23:15:16.309719622 +0000 UTC m=+535.453549827" Oct 30 23:15:16.425422 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:16.424499 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-595664bccf-bmvjk" podStartSLOduration=1.9509055339999999 podStartE2EDuration="3.424480437s" totalImagesPullingTime="1.473574903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:15:16.419906454 +0000 UTC m=+535.563736685" watchObservedRunningTime="2026-10-30 23:15:16.424480437 +0000 UTC m=+535.568310644" Oct 30 23:15:16.432069 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:16.428410 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6948467c5c-czqzk" Oct 30 23:15:16.453823 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:16.453368 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qxr4b"] Oct 30 23:15:16.453823 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:16.453594 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-qxr4b" podUID="fb5cde51-c849-4371-9e64-fbfa54e26e35" containerName="authorino" containerID="cri-o://2c8d051c984326c49585f72e8219d61d3bd15b091e5cfd1ccb4c2d50511a1ad9" gracePeriod=30 Oct 30 23:15:16.470457 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:16.470246 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-9s6ft" Oct 30 23:15:16.552396 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:16.552356 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8d8cdcb-8d67-425b-aa72-c9a4aef16479-kube-api-access-26zcf\" (UniqueName: \"kubernetes.io/projected/b8d8cdcb-8d67-425b-aa72-c9a4aef16479-kube-api-access-26zcf\") pod \"b8d8cdcb-8d67-425b-aa72-c9a4aef16479\" (UID: \"b8d8cdcb-8d67-425b-aa72-c9a4aef16479\") " Oct 30 23:15:16.552611 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:16.552466 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62795711-cd58-437f-b82c-25dd7f019d04-kube-api-access-kzgcf\" (UniqueName: \"kubernetes.io/projected/62795711-cd58-437f-b82c-25dd7f019d04-kube-api-access-kzgcf\") pod \"62795711-cd58-437f-b82c-25dd7f019d04\" (UID: \"62795711-cd58-437f-b82c-25dd7f019d04\") " Oct 30 23:15:16.555339 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:16.555305 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62795711-cd58-437f-b82c-25dd7f019d04-kube-api-access-kzgcf" pod "62795711-cd58-437f-b82c-25dd7f019d04" (UID: "62795711-cd58-437f-b82c-25dd7f019d04"). InnerVolumeSpecName "kube-api-access-kzgcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:15:16.555831 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:16.555804 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d8cdcb-8d67-425b-aa72-c9a4aef16479-kube-api-access-26zcf" pod "b8d8cdcb-8d67-425b-aa72-c9a4aef16479" (UID: "b8d8cdcb-8d67-425b-aa72-c9a4aef16479"). InnerVolumeSpecName "kube-api-access-26zcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:15:16.654181 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:16.654150 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzgcf\" (UniqueName: \"kubernetes.io/projected/62795711-cd58-437f-b82c-25dd7f019d04-kube-api-access-kzgcf\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:15:16.654181 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:16.654178 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26zcf\" (UniqueName: \"kubernetes.io/projected/b8d8cdcb-8d67-425b-aa72-c9a4aef16479-kube-api-access-26zcf\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:15:16.771335 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:16.771308 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-qxr4b" Oct 30 23:15:16.855462 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:16.855432 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb5cde51-c849-4371-9e64-fbfa54e26e35-kube-api-access-7bd4s\" (UniqueName: \"kubernetes.io/projected/fb5cde51-c849-4371-9e64-fbfa54e26e35-kube-api-access-7bd4s\") pod \"fb5cde51-c849-4371-9e64-fbfa54e26e35\" (UID: \"fb5cde51-c849-4371-9e64-fbfa54e26e35\") " Oct 30 23:15:16.857655 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:16.857585 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb5cde51-c849-4371-9e64-fbfa54e26e35-kube-api-access-7bd4s" pod "fb5cde51-c849-4371-9e64-fbfa54e26e35" (UID: "fb5cde51-c849-4371-9e64-fbfa54e26e35"). InnerVolumeSpecName "kube-api-access-7bd4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:15:16.956773 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:16.956740 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bd4s\" (UniqueName: \"kubernetes.io/projected/fb5cde51-c849-4371-9e64-fbfa54e26e35-kube-api-access-7bd4s\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:15:17.081063 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:17.081013 4952 generic.go:356] "Generic (PLEG): container finished" podID="62795711-cd58-437f-b82c-25dd7f019d04" containerID="5de0030c898031d37f60883a34e6769b9b55c4f000c6ae9d6aea66090667ada7" exitCode=0 Oct 30 23:15:17.081525 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:17.081081 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6948467c5c-czqzk" Oct 30 23:15:17.081525 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:17.081100 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6948467c5c-czqzk" event={"ID":"62795711-cd58-437f-b82c-25dd7f019d04","Type":"ContainerDied","Data":"5de0030c898031d37f60883a34e6769b9b55c4f000c6ae9d6aea66090667ada7"} Oct 30 23:15:17.081525 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:17.081157 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6948467c5c-czqzk" event={"ID":"62795711-cd58-437f-b82c-25dd7f019d04","Type":"ContainerDied","Data":"5db53afcd84a08deac22e8ccce768ffbecc3d35b07f13d7ccee1e97de047485d"} Oct 30 23:15:17.081525 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:17.081185 4952 scope.go:122] "RemoveContainer" containerID="5de0030c898031d37f60883a34e6769b9b55c4f000c6ae9d6aea66090667ada7" Oct 30 23:15:17.082703 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:17.082677 4952 generic.go:356] "Generic (PLEG): container finished" podID="b8d8cdcb-8d67-425b-aa72-c9a4aef16479" containerID="b36878723293d90414c50d45ca4f003c005d4c737637c19ceb2519e91a97332f" exitCode=0 Oct 30 23:15:17.082829 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:17.082725 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-9s6ft" Oct 30 23:15:17.082829 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:17.082747 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-9s6ft" event={"ID":"b8d8cdcb-8d67-425b-aa72-c9a4aef16479","Type":"ContainerDied","Data":"b36878723293d90414c50d45ca4f003c005d4c737637c19ceb2519e91a97332f"} Oct 30 23:15:17.082829 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:17.082779 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-9s6ft" event={"ID":"b8d8cdcb-8d67-425b-aa72-c9a4aef16479","Type":"ContainerDied","Data":"aa776ac08e984934e7187841615a60644d31b64c576d9365d2907347130f55d3"} Oct 30 23:15:17.084410 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:17.084387 4952 generic.go:356] "Generic (PLEG): container finished" podID="fb5cde51-c849-4371-9e64-fbfa54e26e35" containerID="2c8d051c984326c49585f72e8219d61d3bd15b091e5cfd1ccb4c2d50511a1ad9" exitCode=0 Oct 30 23:15:17.084520 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:17.084446 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-qxr4b" Oct 30 23:15:17.084520 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:17.084467 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-qxr4b" event={"ID":"fb5cde51-c849-4371-9e64-fbfa54e26e35","Type":"ContainerDied","Data":"2c8d051c984326c49585f72e8219d61d3bd15b091e5cfd1ccb4c2d50511a1ad9"} Oct 30 23:15:17.084520 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:17.084503 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-qxr4b" event={"ID":"fb5cde51-c849-4371-9e64-fbfa54e26e35","Type":"ContainerDied","Data":"a3eaed066c0c51fee61c54d9ef088f8f26ff4ba25aff9add2c316cfd73eb1bcf"} Oct 30 23:15:17.091497 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:17.091480 4952 scope.go:122] "RemoveContainer" containerID="5de0030c898031d37f60883a34e6769b9b55c4f000c6ae9d6aea66090667ada7" Oct 30 23:15:17.091766 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:15:17.091749 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5de0030c898031d37f60883a34e6769b9b55c4f000c6ae9d6aea66090667ada7\": container with ID starting with 5de0030c898031d37f60883a34e6769b9b55c4f000c6ae9d6aea66090667ada7 not found: ID does not exist" containerID="5de0030c898031d37f60883a34e6769b9b55c4f000c6ae9d6aea66090667ada7" Oct 30 23:15:17.091819 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:17.091774 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5de0030c898031d37f60883a34e6769b9b55c4f000c6ae9d6aea66090667ada7"} err="failed to get container status \"5de0030c898031d37f60883a34e6769b9b55c4f000c6ae9d6aea66090667ada7\": rpc error: code = NotFound desc = could not find container \"5de0030c898031d37f60883a34e6769b9b55c4f000c6ae9d6aea66090667ada7\": container with ID starting with 5de0030c898031d37f60883a34e6769b9b55c4f000c6ae9d6aea66090667ada7 not found: ID does not exist" Oct 30 23:15:17.091819 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:17.091789 4952 scope.go:122] "RemoveContainer" containerID="b36878723293d90414c50d45ca4f003c005d4c737637c19ceb2519e91a97332f" Oct 30 23:15:17.099933 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:17.099912 4952 scope.go:122] "RemoveContainer" containerID="b36878723293d90414c50d45ca4f003c005d4c737637c19ceb2519e91a97332f" Oct 30 23:15:17.100214 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:15:17.100194 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b36878723293d90414c50d45ca4f003c005d4c737637c19ceb2519e91a97332f\": container with ID starting with b36878723293d90414c50d45ca4f003c005d4c737637c19ceb2519e91a97332f not found: ID does not exist" containerID="b36878723293d90414c50d45ca4f003c005d4c737637c19ceb2519e91a97332f" Oct 30 23:15:17.100272 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:17.100218 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b36878723293d90414c50d45ca4f003c005d4c737637c19ceb2519e91a97332f"} err="failed to get container status \"b36878723293d90414c50d45ca4f003c005d4c737637c19ceb2519e91a97332f\": rpc error: code = NotFound desc = could not find container \"b36878723293d90414c50d45ca4f003c005d4c737637c19ceb2519e91a97332f\": container with ID starting with b36878723293d90414c50d45ca4f003c005d4c737637c19ceb2519e91a97332f not found: ID does not exist" Oct 30 23:15:17.100272 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:17.100232 4952 scope.go:122] "RemoveContainer" containerID="2c8d051c984326c49585f72e8219d61d3bd15b091e5cfd1ccb4c2d50511a1ad9" Oct 30 23:15:17.107985 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:17.107967 4952 scope.go:122] "RemoveContainer" containerID="2c8d051c984326c49585f72e8219d61d3bd15b091e5cfd1ccb4c2d50511a1ad9" Oct 30 23:15:17.108293 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:15:17.108274 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c8d051c984326c49585f72e8219d61d3bd15b091e5cfd1ccb4c2d50511a1ad9\": container with ID starting with 2c8d051c984326c49585f72e8219d61d3bd15b091e5cfd1ccb4c2d50511a1ad9 not found: ID does not exist" containerID="2c8d051c984326c49585f72e8219d61d3bd15b091e5cfd1ccb4c2d50511a1ad9" Oct 30 23:15:17.108375 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:17.108299 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c8d051c984326c49585f72e8219d61d3bd15b091e5cfd1ccb4c2d50511a1ad9"} err="failed to get container status \"2c8d051c984326c49585f72e8219d61d3bd15b091e5cfd1ccb4c2d50511a1ad9\": rpc error: code = NotFound desc = could not find container \"2c8d051c984326c49585f72e8219d61d3bd15b091e5cfd1ccb4c2d50511a1ad9\": container with ID starting with 2c8d051c984326c49585f72e8219d61d3bd15b091e5cfd1ccb4c2d50511a1ad9 not found: ID does not exist" Oct 30 23:15:17.468810 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:17.249006 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6948467c5c-czqzk"] Oct 30 23:15:17.468810 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:17.254986 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6948467c5c-czqzk"] Oct 30 23:15:17.468810 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:17.364834 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qxr4b"] Oct 30 23:15:17.468810 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:17.368410 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qxr4b"] Oct 30 23:15:17.480147 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:17.480108 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9s6ft"] Oct 30 23:15:17.482607 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:17.482585 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9s6ft"] Oct 30 23:15:18.091936 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:18.091905 4952 generic.go:356] "Generic (PLEG): container finished" podID="f5421fcf-649c-477a-b487-68c52d23f09b" containerID="28fa26b49afd6429443af96350c60890fd9066ae78f90914ebc413ab4f06a79a" exitCode=0 Oct 30 23:15:18.092491 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:18.091982 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" event={"ID":"f5421fcf-649c-477a-b487-68c52d23f09b","Type":"ContainerDied","Data":"28fa26b49afd6429443af96350c60890fd9066ae78f90914ebc413ab4f06a79a"} Oct 30 23:15:18.092491 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:18.092391 4952 scope.go:122] "RemoveContainer" containerID="28fa26b49afd6429443af96350c60890fd9066ae78f90914ebc413ab4f06a79a" Oct 30 23:15:18.093502 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:18.093467 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:15:18.093603 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:18.093546 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:15:18.093603 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:18.093581 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:15:18.992714 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:18.992676 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62795711-cd58-437f-b82c-25dd7f019d04" path="/var/lib/kubelet/pods/62795711-cd58-437f-b82c-25dd7f019d04/volumes" Oct 30 23:15:18.993016 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:18.993002 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8d8cdcb-8d67-425b-aa72-c9a4aef16479" path="/var/lib/kubelet/pods/b8d8cdcb-8d67-425b-aa72-c9a4aef16479/volumes" Oct 30 23:15:18.993385 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:18.993370 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb5cde51-c849-4371-9e64-fbfa54e26e35" path="/var/lib/kubelet/pods/fb5cde51-c849-4371-9e64-fbfa54e26e35/volumes" Oct 30 23:15:19.101799 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:19.101765 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" event={"ID":"f5421fcf-649c-477a-b487-68c52d23f09b","Type":"ContainerStarted","Data":"81a33c2ddbc2daa6cdb23d0ad1fcb15970181cf02e7b29fb21eebae0156621bc"} Oct 30 23:15:19.690064 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:19.682592 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:15:19.690064 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:19.682714 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-dz26x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 23:15:19.690064 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:19.682771 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" podUID="f5421fcf-649c-477a-b487-68c52d23f09b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 23:15:20.690426 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:20.690389 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-dz26x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 23:15:20.691021 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:20.690986 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" podUID="f5421fcf-649c-477a-b487-68c52d23f09b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 23:15:21.141623 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:15:21.141588 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:15:21.486170 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:21.486132 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:15:21.686061 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:21.685399 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-dz26x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 23:15:21.686061 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:21.685457 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" podUID="f5421fcf-649c-477a-b487-68c52d23f09b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 23:15:22.685067 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:22.684392 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-dz26x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 23:15:22.685067 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:22.684454 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" podUID="f5421fcf-649c-477a-b487-68c52d23f09b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 23:15:23.682460 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:23.682418 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-dz26x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 23:15:23.683061 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:23.682769 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" podUID="f5421fcf-649c-477a-b487-68c52d23f09b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 23:15:24.685636 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:24.685608 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:15:24.686627 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:24.686608 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" Oct 30 23:15:26.960256 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:26.960222 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:15:51.141541 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:15:51.141501 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:15:57.718574 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:57.718541 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-p47kn"] Oct 30 23:15:57.718924 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:57.718911 4952 cpu_manager.go:414] "containerMap: removing container" podUID="fb5cde51-c849-4371-9e64-fbfa54e26e35" containerName="authorino" Oct 30 23:15:57.718924 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:57.718923 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb5cde51-c849-4371-9e64-fbfa54e26e35" containerName="authorino" Oct 30 23:15:57.719013 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:57.718948 4952 cpu_manager.go:414] "containerMap: removing container" podUID="b8d8cdcb-8d67-425b-aa72-c9a4aef16479" containerName="authorino" Oct 30 23:15:57.719013 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:57.718953 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8d8cdcb-8d67-425b-aa72-c9a4aef16479" containerName="authorino" Oct 30 23:15:57.719013 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:57.718968 4952 cpu_manager.go:414] "containerMap: removing container" podUID="62795711-cd58-437f-b82c-25dd7f019d04" containerName="authorino" Oct 30 23:15:57.719013 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:57.718972 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62795711-cd58-437f-b82c-25dd7f019d04" containerName="authorino" Oct 30 23:15:57.719194 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:57.719030 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8d8cdcb-8d67-425b-aa72-c9a4aef16479" containerName="authorino" Oct 30 23:15:57.719194 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:57.719056 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="62795711-cd58-437f-b82c-25dd7f019d04" containerName="authorino" Oct 30 23:15:57.719194 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:57.719065 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb5cde51-c849-4371-9e64-fbfa54e26e35" containerName="authorino" Oct 30 23:15:57.721303 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:57.721287 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-p47kn" Oct 30 23:15:57.778600 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:57.778575 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-p47kn"] Oct 30 23:15:57.819806 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:57.819780 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/656eacbe-1345-4cdd-b7c7-efa60412553d-cert\") pod \"llmisvc-controller-manager-6466c554d8-p47kn\" (UID: \"656eacbe-1345-4cdd-b7c7-efa60412553d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-p47kn" Oct 30 23:15:57.819926 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:57.819816 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdppz\" (UniqueName: \"kubernetes.io/projected/656eacbe-1345-4cdd-b7c7-efa60412553d-kube-api-access-qdppz\") pod \"llmisvc-controller-manager-6466c554d8-p47kn\" (UID: \"656eacbe-1345-4cdd-b7c7-efa60412553d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-p47kn" Oct 30 23:15:57.920509 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:57.920483 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/656eacbe-1345-4cdd-b7c7-efa60412553d-cert\") pod \"llmisvc-controller-manager-6466c554d8-p47kn\" (UID: \"656eacbe-1345-4cdd-b7c7-efa60412553d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-p47kn" Oct 30 23:15:57.920671 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:57.920523 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdppz\" (UniqueName: \"kubernetes.io/projected/656eacbe-1345-4cdd-b7c7-efa60412553d-kube-api-access-qdppz\") pod \"llmisvc-controller-manager-6466c554d8-p47kn\" (UID: \"656eacbe-1345-4cdd-b7c7-efa60412553d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-p47kn" Oct 30 23:15:57.923085 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:57.923060 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/656eacbe-1345-4cdd-b7c7-efa60412553d-cert\") pod \"llmisvc-controller-manager-6466c554d8-p47kn\" (UID: \"656eacbe-1345-4cdd-b7c7-efa60412553d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-p47kn" Oct 30 23:15:57.977735 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:57.977676 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdppz\" (UniqueName: \"kubernetes.io/projected/656eacbe-1345-4cdd-b7c7-efa60412553d-kube-api-access-qdppz\") pod \"llmisvc-controller-manager-6466c554d8-p47kn\" (UID: \"656eacbe-1345-4cdd-b7c7-efa60412553d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-p47kn" Oct 30 23:15:58.031892 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:58.031870 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-p47kn" Oct 30 23:15:58.401562 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:58.401537 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-p47kn"] Oct 30 23:15:58.402859 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:15:58.402833 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod656eacbe_1345_4cdd_b7c7_efa60412553d.slice/crio-b8cfc748942c82590ee2cce83b8b2c28c081f8581c45de649bea06962f0d42ea WatchSource:0}: Error finding container b8cfc748942c82590ee2cce83b8b2c28c081f8581c45de649bea06962f0d42ea: Status 404 returned error can't find the container with id b8cfc748942c82590ee2cce83b8b2c28c081f8581c45de649bea06962f0d42ea Oct 30 23:15:59.319824 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:59.319787 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-p47kn" event={"ID":"656eacbe-1345-4cdd-b7c7-efa60412553d","Type":"ContainerStarted","Data":"24bf7501b3b072adad337ec784e4d04364b14e7b386fa5f6e2cd124648dd1eb2"} Oct 30 23:15:59.320222 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:59.319849 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-p47kn" event={"ID":"656eacbe-1345-4cdd-b7c7-efa60412553d","Type":"ContainerStarted","Data":"b8cfc748942c82590ee2cce83b8b2c28c081f8581c45de649bea06962f0d42ea"} Oct 30 23:15:59.320222 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:59.319880 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-p47kn" Oct 30 23:15:59.434976 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:15:59.434887 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-p47kn" podStartSLOduration=1.6775144210000001 podStartE2EDuration="2.434873432s" totalImagesPullingTime="757.359011ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:15:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:15:59.433060789 +0000 UTC m=+578.576891014" watchObservedRunningTime="2026-10-30 23:15:59.434873432 +0000 UTC m=+578.578703652" Oct 30 23:16:13.851591 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:16:13.851552 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:16:21.013367 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:16:21.013336 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft_91651ce2-4673-4ced-b26d-e94ff8829d11/extract/0.log" Oct 30 23:16:21.014144 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:16:21.014118 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft_91651ce2-4673-4ced-b26d-e94ff8829d11/extract/0.log" Oct 30 23:16:21.023184 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:16:21.023165 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/2.log" Oct 30 23:16:21.023849 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:16:21.023832 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/2.log" Oct 30 23:16:21.035564 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:16:21.035541 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 30 23:16:21.036164 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:16:21.036141 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 30 23:16:21.141736 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:16:21.141696 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:16:30.330570 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:16:30.330541 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-p47kn" Oct 30 23:16:30.496910 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:16:30.496876 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-qc829"] Oct 30 23:16:30.497211 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:16:30.497162 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qc829" podUID="eb357cbd-85f5-4057-b7bb-852fb8465862" containerName="manager" containerID="cri-o://aa69bb0b21819fb80129046f4dea82769b290051b4ce21a70f75879fca48c73b" gracePeriod=30 Oct 30 23:16:30.745631 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:16:30.745609 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qc829" Oct 30 23:16:30.785978 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:16:30.785950 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb357cbd-85f5-4057-b7bb-852fb8465862-kube-api-access-z5rf5\" (UniqueName: \"kubernetes.io/projected/eb357cbd-85f5-4057-b7bb-852fb8465862-kube-api-access-z5rf5\") pod \"eb357cbd-85f5-4057-b7bb-852fb8465862\" (UID: \"eb357cbd-85f5-4057-b7bb-852fb8465862\") " Oct 30 23:16:30.786153 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:16:30.785997 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb357cbd-85f5-4057-b7bb-852fb8465862-cert\" (UniqueName: \"kubernetes.io/secret/eb357cbd-85f5-4057-b7bb-852fb8465862-cert\") pod \"eb357cbd-85f5-4057-b7bb-852fb8465862\" (UID: \"eb357cbd-85f5-4057-b7bb-852fb8465862\") " Oct 30 23:16:30.788190 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:16:30.788166 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb357cbd-85f5-4057-b7bb-852fb8465862-cert" pod "eb357cbd-85f5-4057-b7bb-852fb8465862" (UID: "eb357cbd-85f5-4057-b7bb-852fb8465862"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:16:30.788270 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:16:30.788214 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb357cbd-85f5-4057-b7bb-852fb8465862-kube-api-access-z5rf5" pod "eb357cbd-85f5-4057-b7bb-852fb8465862" (UID: "eb357cbd-85f5-4057-b7bb-852fb8465862"). InnerVolumeSpecName "kube-api-access-z5rf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:16:30.887146 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:16:30.887071 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5rf5\" (UniqueName: \"kubernetes.io/projected/eb357cbd-85f5-4057-b7bb-852fb8465862-kube-api-access-z5rf5\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:16:30.887146 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:16:30.887098 4952 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb357cbd-85f5-4057-b7bb-852fb8465862-cert\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:16:31.407186 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:16:31.407153 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:16:31.496261 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:16:31.496224 4952 generic.go:356] "Generic (PLEG): container finished" podID="eb357cbd-85f5-4057-b7bb-852fb8465862" containerID="aa69bb0b21819fb80129046f4dea82769b290051b4ce21a70f75879fca48c73b" exitCode=0 Oct 30 23:16:31.496261 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:16:31.496267 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qc829" Oct 30 23:16:31.496519 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:16:31.496291 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qc829" event={"ID":"eb357cbd-85f5-4057-b7bb-852fb8465862","Type":"ContainerDied","Data":"aa69bb0b21819fb80129046f4dea82769b290051b4ce21a70f75879fca48c73b"} Oct 30 23:16:31.496519 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:16:31.496353 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qc829" event={"ID":"eb357cbd-85f5-4057-b7bb-852fb8465862","Type":"ContainerDied","Data":"9397ab52de0a4eb643f6c41f84286295685ab71f58fbd46c50141c8c0837b79e"} Oct 30 23:16:31.496519 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:16:31.496383 4952 scope.go:122] "RemoveContainer" containerID="aa69bb0b21819fb80129046f4dea82769b290051b4ce21a70f75879fca48c73b" Oct 30 23:16:31.505893 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:16:31.505695 4952 scope.go:122] "RemoveContainer" containerID="aa69bb0b21819fb80129046f4dea82769b290051b4ce21a70f75879fca48c73b" Oct 30 23:16:31.505978 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:16:31.505958 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa69bb0b21819fb80129046f4dea82769b290051b4ce21a70f75879fca48c73b\": container with ID starting with aa69bb0b21819fb80129046f4dea82769b290051b4ce21a70f75879fca48c73b not found: ID does not exist" containerID="aa69bb0b21819fb80129046f4dea82769b290051b4ce21a70f75879fca48c73b" Oct 30 23:16:31.506018 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:16:31.505986 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa69bb0b21819fb80129046f4dea82769b290051b4ce21a70f75879fca48c73b"} err="failed to get container status \"aa69bb0b21819fb80129046f4dea82769b290051b4ce21a70f75879fca48c73b\": rpc error: code = NotFound desc = could not find container \"aa69bb0b21819fb80129046f4dea82769b290051b4ce21a70f75879fca48c73b\": container with ID starting with aa69bb0b21819fb80129046f4dea82769b290051b4ce21a70f75879fca48c73b not found: ID does not exist" Oct 30 23:16:31.665525 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:16:31.665438 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-qc829"] Oct 30 23:16:31.672292 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:16:31.672269 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-qc829"] Oct 30 23:16:31.672913 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:16:31.672854 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:16:32.969189 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:16:32.969157 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb357cbd-85f5-4057-b7bb-852fb8465862" path="/var/lib/kubelet/pods/eb357cbd-85f5-4057-b7bb-852fb8465862/volumes" Oct 30 23:16:35.444600 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:16:35.444563 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:16:51.142275 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:16:51.142233 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:16:52.863715 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:16:52.863684 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:16:56.241977 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:16:56.241947 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:16:56.352792 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:16:56.352759 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:16:59.464215 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:16:59.464182 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:17:03.313771 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:03.313735 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:17:06.794667 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:06.794628 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:17:09.212751 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:09.212716 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:17:13.763954 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:13.763917 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:17:14.110540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:14.110511 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:17:17.315602 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:17.315562 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:17:19.126498 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:19.126466 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:17:20.648414 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:20.648382 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:17:21.141224 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:17:21.141192 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:17:24.445167 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:24.445121 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:17:36.570011 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:36.569926 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-cf46b9688-9vzlq"] Oct 30 23:17:36.570584 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:36.570564 4952 cpu_manager.go:414] "containerMap: removing container" podUID="eb357cbd-85f5-4057-b7bb-852fb8465862" containerName="manager" Oct 30 23:17:36.570631 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:36.570587 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb357cbd-85f5-4057-b7bb-852fb8465862" containerName="manager" Oct 30 23:17:36.570702 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:36.570689 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb357cbd-85f5-4057-b7bb-852fb8465862" containerName="manager" Oct 30 23:17:36.573861 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:36.573842 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cf46b9688-9vzlq" Oct 30 23:17:36.633581 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:36.633557 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-cf46b9688-9vzlq"] Oct 30 23:17:36.727198 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:36.727160 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cn65\" (UniqueName: \"kubernetes.io/projected/ee7ad485-5d24-49cc-8072-0211b6ace77f-kube-api-access-5cn65\") pod \"authorino-cf46b9688-9vzlq\" (UID: \"ee7ad485-5d24-49cc-8072-0211b6ace77f\") " pod="kuadrant-system/authorino-cf46b9688-9vzlq" Oct 30 23:17:36.727368 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:36.727208 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ee7ad485-5d24-49cc-8072-0211b6ace77f-tls-cert\") pod \"authorino-cf46b9688-9vzlq\" (UID: \"ee7ad485-5d24-49cc-8072-0211b6ace77f\") " pod="kuadrant-system/authorino-cf46b9688-9vzlq" Oct 30 23:17:36.828421 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:36.828342 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cn65\" (UniqueName: \"kubernetes.io/projected/ee7ad485-5d24-49cc-8072-0211b6ace77f-kube-api-access-5cn65\") pod \"authorino-cf46b9688-9vzlq\" (UID: \"ee7ad485-5d24-49cc-8072-0211b6ace77f\") " pod="kuadrant-system/authorino-cf46b9688-9vzlq" Oct 30 23:17:36.828421 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:36.828390 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ee7ad485-5d24-49cc-8072-0211b6ace77f-tls-cert\") pod \"authorino-cf46b9688-9vzlq\" (UID: \"ee7ad485-5d24-49cc-8072-0211b6ace77f\") " pod="kuadrant-system/authorino-cf46b9688-9vzlq" Oct 30 23:17:36.831173 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:36.831146 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ee7ad485-5d24-49cc-8072-0211b6ace77f-tls-cert\") pod \"authorino-cf46b9688-9vzlq\" (UID: \"ee7ad485-5d24-49cc-8072-0211b6ace77f\") " pod="kuadrant-system/authorino-cf46b9688-9vzlq" Oct 30 23:17:36.875558 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:36.875522 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:17:36.886527 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:36.886494 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cn65\" (UniqueName: \"kubernetes.io/projected/ee7ad485-5d24-49cc-8072-0211b6ace77f-kube-api-access-5cn65\") pod \"authorino-cf46b9688-9vzlq\" (UID: \"ee7ad485-5d24-49cc-8072-0211b6ace77f\") " pod="kuadrant-system/authorino-cf46b9688-9vzlq" Oct 30 23:17:37.136296 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:37.136262 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:17:37.185551 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:37.185524 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cf46b9688-9vzlq" Oct 30 23:17:37.556682 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:17:37.556645 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee7ad485_5d24_49cc_8072_0211b6ace77f.slice/crio-f5371949cae88223c4e30860a919acf365ad629eb35c94b1c8d7dbd4e8a6643c WatchSource:0}: Error finding container f5371949cae88223c4e30860a919acf365ad629eb35c94b1c8d7dbd4e8a6643c: Status 404 returned error can't find the container with id f5371949cae88223c4e30860a919acf365ad629eb35c94b1c8d7dbd4e8a6643c Oct 30 23:17:37.558247 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:37.558227 4952 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 23:17:37.558822 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:37.558801 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-cf46b9688-9vzlq"] Oct 30 23:17:37.873366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:37.873319 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cf46b9688-9vzlq" event={"ID":"ee7ad485-5d24-49cc-8072-0211b6ace77f","Type":"ContainerStarted","Data":"f5371949cae88223c4e30860a919acf365ad629eb35c94b1c8d7dbd4e8a6643c"} Oct 30 23:17:39.885775 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:39.885726 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cf46b9688-9vzlq" event={"ID":"ee7ad485-5d24-49cc-8072-0211b6ace77f","Type":"ContainerStarted","Data":"d37f742301553aa7917c11da918e8b9e452b83febf13eb5dd8e2824a6bc16cb4"} Oct 30 23:17:40.002141 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:40.002084 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-cf46b9688-9vzlq" podStartSLOduration=2.5356375399999997 podStartE2EDuration="4.002066003s" totalImagesPullingTime="1.466428463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:17:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:17:39.998101203 +0000 UTC m=+679.141931426" watchObservedRunningTime="2026-10-30 23:17:40.002066003 +0000 UTC m=+679.145896210" Oct 30 23:17:40.050133 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:40.050101 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-595664bccf-bmvjk"] Oct 30 23:17:40.050306 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:40.050282 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-595664bccf-bmvjk" podUID="7a0b5138-0233-4c56-b97d-6a68d2d153f2" containerName="authorino" containerID="cri-o://26309694a2fab6813db6865adfa4f1ecf943f0a3f5970825e165b5bc628a1063" gracePeriod=30 Oct 30 23:17:40.315686 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:40.315660 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-595664bccf-bmvjk" Oct 30 23:17:40.458839 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:40.458752 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a0b5138-0233-4c56-b97d-6a68d2d153f2-tls-cert\" (UniqueName: \"kubernetes.io/secret/7a0b5138-0233-4c56-b97d-6a68d2d153f2-tls-cert\") pod \"7a0b5138-0233-4c56-b97d-6a68d2d153f2\" (UID: \"7a0b5138-0233-4c56-b97d-6a68d2d153f2\") " Oct 30 23:17:40.458839 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:40.458827 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a0b5138-0233-4c56-b97d-6a68d2d153f2-kube-api-access-mcrzm\" (UniqueName: \"kubernetes.io/projected/7a0b5138-0233-4c56-b97d-6a68d2d153f2-kube-api-access-mcrzm\") pod \"7a0b5138-0233-4c56-b97d-6a68d2d153f2\" (UID: \"7a0b5138-0233-4c56-b97d-6a68d2d153f2\") " Oct 30 23:17:40.461022 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:40.460987 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a0b5138-0233-4c56-b97d-6a68d2d153f2-kube-api-access-mcrzm" pod "7a0b5138-0233-4c56-b97d-6a68d2d153f2" (UID: "7a0b5138-0233-4c56-b97d-6a68d2d153f2"). InnerVolumeSpecName "kube-api-access-mcrzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:17:40.470663 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:40.470629 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a0b5138-0233-4c56-b97d-6a68d2d153f2-tls-cert" pod "7a0b5138-0233-4c56-b97d-6a68d2d153f2" (UID: "7a0b5138-0233-4c56-b97d-6a68d2d153f2"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:17:40.560459 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:40.560426 4952 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7a0b5138-0233-4c56-b97d-6a68d2d153f2-tls-cert\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:17:40.560459 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:40.560458 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcrzm\" (UniqueName: \"kubernetes.io/projected/7a0b5138-0233-4c56-b97d-6a68d2d153f2-kube-api-access-mcrzm\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:17:40.893054 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:40.893004 4952 generic.go:356] "Generic (PLEG): container finished" podID="7a0b5138-0233-4c56-b97d-6a68d2d153f2" containerID="26309694a2fab6813db6865adfa4f1ecf943f0a3f5970825e165b5bc628a1063" exitCode=0 Oct 30 23:17:40.893578 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:40.893077 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-595664bccf-bmvjk" Oct 30 23:17:40.893578 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:40.893085 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-595664bccf-bmvjk" event={"ID":"7a0b5138-0233-4c56-b97d-6a68d2d153f2","Type":"ContainerDied","Data":"26309694a2fab6813db6865adfa4f1ecf943f0a3f5970825e165b5bc628a1063"} Oct 30 23:17:40.893578 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:40.893128 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-595664bccf-bmvjk" event={"ID":"7a0b5138-0233-4c56-b97d-6a68d2d153f2","Type":"ContainerDied","Data":"9295c4ab468dfe2d7eee005442431a7b7a54b73864fe2a965a4500da1fb703a0"} Oct 30 23:17:40.893578 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:40.893152 4952 scope.go:122] "RemoveContainer" containerID="26309694a2fab6813db6865adfa4f1ecf943f0a3f5970825e165b5bc628a1063" Oct 30 23:17:40.904338 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:40.904320 4952 scope.go:122] "RemoveContainer" containerID="26309694a2fab6813db6865adfa4f1ecf943f0a3f5970825e165b5bc628a1063" Oct 30 23:17:40.904646 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:17:40.904623 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26309694a2fab6813db6865adfa4f1ecf943f0a3f5970825e165b5bc628a1063\": container with ID starting with 26309694a2fab6813db6865adfa4f1ecf943f0a3f5970825e165b5bc628a1063 not found: ID does not exist" containerID="26309694a2fab6813db6865adfa4f1ecf943f0a3f5970825e165b5bc628a1063" Oct 30 23:17:40.904749 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:40.904655 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26309694a2fab6813db6865adfa4f1ecf943f0a3f5970825e165b5bc628a1063"} err="failed to get container status \"26309694a2fab6813db6865adfa4f1ecf943f0a3f5970825e165b5bc628a1063\": rpc error: code = NotFound desc = could not find container \"26309694a2fab6813db6865adfa4f1ecf943f0a3f5970825e165b5bc628a1063\": container with ID starting with 26309694a2fab6813db6865adfa4f1ecf943f0a3f5970825e165b5bc628a1063 not found: ID does not exist" Oct 30 23:17:41.059099 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:41.059061 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-595664bccf-bmvjk"] Oct 30 23:17:41.063113 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:41.063087 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-595664bccf-bmvjk"] Oct 30 23:17:42.969768 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:42.969734 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a0b5138-0233-4c56-b97d-6a68d2d153f2" path="/var/lib/kubelet/pods/7a0b5138-0233-4c56-b97d-6a68d2d153f2/volumes" Oct 30 23:17:43.999405 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:43.999366 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:17:44.220403 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:17:44.220370 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:17:51.141301 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:17:51.141268 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:18:09.299889 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:09.299859 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:09.774247 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:09.774214 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:11.259242 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:11.259205 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:11.461096 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:11.461067 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:12.853831 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:12.853796 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:13.069052 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:13.068970 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:14.462576 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:14.462539 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:21.143747 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:18:21.143708 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:18:37.392183 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:37.392153 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:37.615624 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:37.615589 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:38.434684 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:38.434650 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:38.738632 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:38.738542 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:38.958472 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:38.958441 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:39.161092 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:39.161052 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:39.374990 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:39.374956 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:39.602704 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:39.602673 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:39.807372 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:39.807328 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:40.026778 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:40.026701 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:40.240723 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:40.240691 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:40.437910 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:40.437881 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:40.644936 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:40.644903 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:43.136131 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:43.136099 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:43.401336 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:43.401268 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:43.843213 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:43.843185 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:44.093340 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:44.093264 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:44.472649 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:44.472577 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:44.721074 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:44.721024 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:45.280654 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:45.280621 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:45.543811 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:45.543727 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:45.936687 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:45.936653 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:46.230251 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:46.230165 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:48.242213 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:48.242183 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:48.566735 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:48.566652 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:49.620625 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:49.620595 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:49.839891 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:49.839854 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:50.137308 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:50.137279 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:50.376177 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:50.376142 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:51.142133 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:18:51.142099 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:18:53.162374 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:53.162344 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:53.385381 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:53.385348 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:55.054691 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:55.054647 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:55.264429 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:55.264392 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:55.478356 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:55.478316 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:55.795545 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:55.795446 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:56.014063 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:56.014011 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:56.114432 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:56.114399 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-aadaa6-85599b78b7-mz9sj"] Oct 30 23:18:56.114906 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:56.114890 4952 cpu_manager.go:414] "containerMap: removing container" podUID="7a0b5138-0233-4c56-b97d-6a68d2d153f2" containerName="authorino" Oct 30 23:18:56.114963 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:56.114909 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a0b5138-0233-4c56-b97d-6a68d2d153f2" containerName="authorino" Oct 30 23:18:56.115006 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:56.114983 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a0b5138-0233-4c56-b97d-6a68d2d153f2" containerName="authorino" Oct 30 23:18:56.118162 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:56.118145 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aadaa6-85599b78b7-mz9sj" Oct 30 23:18:56.170229 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:56.170208 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-aadaa6\"" Oct 30 23:18:56.219725 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:56.219697 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-aadaa6-dockercfg-ktsdn\"" Oct 30 23:18:56.222824 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:56.222801 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-aadaa6-85599b78b7-mz9sj"] Oct 30 23:18:56.273007 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:56.272972 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x56ql\" (UniqueName: \"kubernetes.io/projected/9fa21217-0585-4176-8a68-b7ec85bf1068-kube-api-access-x56ql\") pod \"payload-pre-processing-e2e-worker-w6-aadaa6-85599b78b7-mz9sj\" (UID: \"9fa21217-0585-4176-8a68-b7ec85bf1068\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aadaa6-85599b78b7-mz9sj" Oct 30 23:18:56.273161 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:56.273014 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fa21217-0585-4176-8a68-b7ec85bf1068-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-aadaa6-85599b78b7-mz9sj\" (UID: \"9fa21217-0585-4176-8a68-b7ec85bf1068\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aadaa6-85599b78b7-mz9sj" Oct 30 23:18:56.374055 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:56.373958 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x56ql\" (UniqueName: \"kubernetes.io/projected/9fa21217-0585-4176-8a68-b7ec85bf1068-kube-api-access-x56ql\") pod \"payload-pre-processing-e2e-worker-w6-aadaa6-85599b78b7-mz9sj\" (UID: \"9fa21217-0585-4176-8a68-b7ec85bf1068\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aadaa6-85599b78b7-mz9sj" Oct 30 23:18:56.374055 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:56.374003 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fa21217-0585-4176-8a68-b7ec85bf1068-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-aadaa6-85599b78b7-mz9sj\" (UID: \"9fa21217-0585-4176-8a68-b7ec85bf1068\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aadaa6-85599b78b7-mz9sj" Oct 30 23:18:56.374719 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:56.374694 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fa21217-0585-4176-8a68-b7ec85bf1068-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-aadaa6-85599b78b7-mz9sj\" (UID: \"9fa21217-0585-4176-8a68-b7ec85bf1068\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aadaa6-85599b78b7-mz9sj" Oct 30 23:18:56.432134 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:56.432104 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x56ql\" (UniqueName: \"kubernetes.io/projected/9fa21217-0585-4176-8a68-b7ec85bf1068-kube-api-access-x56ql\") pod \"payload-pre-processing-e2e-worker-w6-aadaa6-85599b78b7-mz9sj\" (UID: \"9fa21217-0585-4176-8a68-b7ec85bf1068\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aadaa6-85599b78b7-mz9sj" Oct 30 23:18:56.731333 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:56.731273 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aadaa6-85599b78b7-mz9sj" Oct 30 23:18:57.103579 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:57.103552 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-aadaa6-85599b78b7-mz9sj"] Oct 30 23:18:57.104726 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:18:57.104691 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fa21217_0585_4176_8a68_b7ec85bf1068.slice/crio-3dc59c3c9588b9e5f72459fe2588b07d2f41bcec05ba18e7749b004e163a66e3 WatchSource:0}: Error finding container 3dc59c3c9588b9e5f72459fe2588b07d2f41bcec05ba18e7749b004e163a66e3: Status 404 returned error can't find the container with id 3dc59c3c9588b9e5f72459fe2588b07d2f41bcec05ba18e7749b004e163a66e3 Oct 30 23:18:57.318805 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:57.318777 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aadaa6-85599b78b7-mz9sj" event={"ID":"9fa21217-0585-4176-8a68-b7ec85bf1068","Type":"ContainerStarted","Data":"3dc59c3c9588b9e5f72459fe2588b07d2f41bcec05ba18e7749b004e163a66e3"} Oct 30 23:18:57.830264 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:57.830230 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:18:58.345587 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:58.345466 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:58.559376 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:58.559341 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:18:58.725174 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:58.725090 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b81a7a-5d9fb6cfb7-6wttp"] Oct 30 23:18:58.730903 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:58.730879 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b81a7a-5d9fb6cfb7-6wttp" Oct 30 23:18:58.736301 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:58.736252 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b81a7a-5d9fb6cfb7-6wttp"] Oct 30 23:18:58.790064 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:58.789991 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-b81a7a-dockercfg-x7klb\"" Oct 30 23:18:58.790247 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:58.790071 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-b81a7a\"" Oct 30 23:18:58.796573 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:58.796544 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa45c557-0216-4475-9a34-e4a75bee03c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-b81a7a-5d9fb6cfb7-6wttp\" (UID: \"aa45c557-0216-4475-9a34-e4a75bee03c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b81a7a-5d9fb6cfb7-6wttp" Oct 30 23:18:58.796716 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:58.796595 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92zgr\" (UniqueName: \"kubernetes.io/projected/aa45c557-0216-4475-9a34-e4a75bee03c3-kube-api-access-92zgr\") pod \"payload-pre-processing-e2e-worker-w1-b81a7a-5d9fb6cfb7-6wttp\" (UID: \"aa45c557-0216-4475-9a34-e4a75bee03c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b81a7a-5d9fb6cfb7-6wttp" Oct 30 23:18:58.884707 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:58.884672 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b81a7a-78746cfd9c-ql62c"] Oct 30 23:18:58.891564 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:58.891540 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b81a7a-78746cfd9c-ql62c" Oct 30 23:18:58.898217 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:58.897913 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa45c557-0216-4475-9a34-e4a75bee03c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-b81a7a-5d9fb6cfb7-6wttp\" (UID: \"aa45c557-0216-4475-9a34-e4a75bee03c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b81a7a-5d9fb6cfb7-6wttp" Oct 30 23:18:58.898217 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:58.897973 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92zgr\" (UniqueName: \"kubernetes.io/projected/aa45c557-0216-4475-9a34-e4a75bee03c3-kube-api-access-92zgr\") pod \"payload-pre-processing-e2e-worker-w1-b81a7a-5d9fb6cfb7-6wttp\" (UID: \"aa45c557-0216-4475-9a34-e4a75bee03c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b81a7a-5d9fb6cfb7-6wttp" Oct 30 23:18:58.898516 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:58.898490 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55a5e296-29c3-49eb-b79f-63378924400a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b81a7a-78746cfd9c-ql62c\" (UID: \"55a5e296-29c3-49eb-b79f-63378924400a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b81a7a-78746cfd9c-ql62c" Oct 30 23:18:58.898652 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:58.898541 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8vdb\" (UniqueName: \"kubernetes.io/projected/55a5e296-29c3-49eb-b79f-63378924400a-kube-api-access-t8vdb\") pod \"payload-processing-e2e-worker-w1-b81a7a-78746cfd9c-ql62c\" (UID: \"55a5e296-29c3-49eb-b79f-63378924400a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b81a7a-78746cfd9c-ql62c" Oct 30 23:18:58.898991 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:58.898970 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa45c557-0216-4475-9a34-e4a75bee03c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-b81a7a-5d9fb6cfb7-6wttp\" (UID: \"aa45c557-0216-4475-9a34-e4a75bee03c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b81a7a-5d9fb6cfb7-6wttp" Oct 30 23:18:58.947671 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:58.947643 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b81a7a-78746cfd9c-ql62c"] Oct 30 23:18:58.958837 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:58.958810 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92zgr\" (UniqueName: \"kubernetes.io/projected/aa45c557-0216-4475-9a34-e4a75bee03c3-kube-api-access-92zgr\") pod \"payload-pre-processing-e2e-worker-w1-b81a7a-5d9fb6cfb7-6wttp\" (UID: \"aa45c557-0216-4475-9a34-e4a75bee03c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b81a7a-5d9fb6cfb7-6wttp" Oct 30 23:18:58.999019 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:58.998953 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55a5e296-29c3-49eb-b79f-63378924400a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b81a7a-78746cfd9c-ql62c\" (UID: \"55a5e296-29c3-49eb-b79f-63378924400a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b81a7a-78746cfd9c-ql62c" Oct 30 23:18:58.999019 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:58.998999 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8vdb\" (UniqueName: \"kubernetes.io/projected/55a5e296-29c3-49eb-b79f-63378924400a-kube-api-access-t8vdb\") pod \"payload-processing-e2e-worker-w1-b81a7a-78746cfd9c-ql62c\" (UID: \"55a5e296-29c3-49eb-b79f-63378924400a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b81a7a-78746cfd9c-ql62c" Oct 30 23:18:58.999785 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:58.999762 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55a5e296-29c3-49eb-b79f-63378924400a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b81a7a-78746cfd9c-ql62c\" (UID: \"55a5e296-29c3-49eb-b79f-63378924400a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b81a7a-78746cfd9c-ql62c" Oct 30 23:18:59.052083 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:59.052059 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b81a7a-5d9fb6cfb7-6wttp" Oct 30 23:18:59.058169 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:59.058110 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8vdb\" (UniqueName: \"kubernetes.io/projected/55a5e296-29c3-49eb-b79f-63378924400a-kube-api-access-t8vdb\") pod \"payload-processing-e2e-worker-w1-b81a7a-78746cfd9c-ql62c\" (UID: \"55a5e296-29c3-49eb-b79f-63378924400a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b81a7a-78746cfd9c-ql62c" Oct 30 23:18:59.206981 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:59.206952 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b81a7a-78746cfd9c-ql62c" Oct 30 23:18:59.438660 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:59.438628 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b81a7a-5d9fb6cfb7-6wttp"] Oct 30 23:18:59.439788 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:18:59.439741 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa45c557_0216_4475_9a34_e4a75bee03c3.slice/crio-11522963f7763ae268f22f833693074cf534966bbbda646d2249056e46a42ecf WatchSource:0}: Error finding container 11522963f7763ae268f22f833693074cf534966bbbda646d2249056e46a42ecf: Status 404 returned error can't find the container with id 11522963f7763ae268f22f833693074cf534966bbbda646d2249056e46a42ecf Oct 30 23:18:59.586363 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:18:59.586331 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b81a7a-78746cfd9c-ql62c"] Oct 30 23:18:59.590869 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:18:59.590839 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55a5e296_29c3_49eb_b79f_63378924400a.slice/crio-ace6a442078120874b6c467f0ebe09b3a81f11d8ef0b76dc5169fd5652e8d247 WatchSource:0}: Error finding container ace6a442078120874b6c467f0ebe09b3a81f11d8ef0b76dc5169fd5652e8d247: Status 404 returned error can't find the container with id ace6a442078120874b6c467f0ebe09b3a81f11d8ef0b76dc5169fd5652e8d247 Oct 30 23:19:00.341191 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:00.341153 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b81a7a-78746cfd9c-ql62c" event={"ID":"55a5e296-29c3-49eb-b79f-63378924400a","Type":"ContainerStarted","Data":"ace6a442078120874b6c467f0ebe09b3a81f11d8ef0b76dc5169fd5652e8d247"} Oct 30 23:19:00.342897 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:00.342866 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b81a7a-5d9fb6cfb7-6wttp" event={"ID":"aa45c557-0216-4475-9a34-e4a75bee03c3","Type":"ContainerStarted","Data":"11522963f7763ae268f22f833693074cf534966bbbda646d2249056e46a42ecf"} Oct 30 23:19:00.344730 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:00.344702 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aadaa6-85599b78b7-mz9sj" event={"ID":"9fa21217-0585-4176-8a68-b7ec85bf1068","Type":"ContainerStarted","Data":"87059b69a8f8a54972b0d94c7d42c78146941c1177c2761a33756828198ad328"} Oct 30 23:19:00.344889 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:00.344876 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aadaa6-85599b78b7-mz9sj" Oct 30 23:19:00.467070 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:00.467003 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aadaa6-85599b78b7-mz9sj" podStartSLOduration=1.859526799 podStartE2EDuration="4.466984706s" totalImagesPullingTime="2.607457907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:00.463607824 +0000 UTC m=+759.607438049" watchObservedRunningTime="2026-10-30 23:19:00.466984706 +0000 UTC m=+759.610814912" Oct 30 23:19:00.592021 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:00.591989 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:00.815652 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:00.815618 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:01.174658 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:01.174627 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:01.284682 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:01.284654 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3c9613-75d9965b8c-g924m"] Oct 30 23:19:01.288416 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:01.288396 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3c9613-75d9965b8c-g924m" Oct 30 23:19:01.316709 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:01.316687 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec586e23-efdd-4fd1-b54c-3c11b5d90ae1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-3c9613-75d9965b8c-g924m\" (UID: \"ec586e23-efdd-4fd1-b54c-3c11b5d90ae1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3c9613-75d9965b8c-g924m" Oct 30 23:19:01.316824 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:01.316731 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srf55\" (UniqueName: \"kubernetes.io/projected/ec586e23-efdd-4fd1-b54c-3c11b5d90ae1-kube-api-access-srf55\") pod \"payload-pre-processing-e2e-worker-w2-3c9613-75d9965b8c-g924m\" (UID: \"ec586e23-efdd-4fd1-b54c-3c11b5d90ae1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3c9613-75d9965b8c-g924m" Oct 30 23:19:01.343243 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:01.343223 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-3c9613\"" Oct 30 23:19:01.343380 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:01.343257 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-3c9613-dockercfg-hb49l\"" Oct 30 23:19:01.345915 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:01.345898 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3c9613-75d9965b8c-g924m"] Oct 30 23:19:01.353701 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:01.353669 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b81a7a-78746cfd9c-ql62c" event={"ID":"55a5e296-29c3-49eb-b79f-63378924400a","Type":"ContainerStarted","Data":"87dbf0767f3c3368685fd177cf912f021d6c3abfd4fcfd31649baf4beef93103"} Oct 30 23:19:01.353827 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:01.353737 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-b81a7a-78746cfd9c-ql62c" Oct 30 23:19:01.355672 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:01.355646 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b81a7a-5d9fb6cfb7-6wttp" event={"ID":"aa45c557-0216-4475-9a34-e4a75bee03c3","Type":"ContainerStarted","Data":"52cb612c1ca2483cd95ca99a8a278ff41240c8ccda628e2f5980676172c94fd3"} Oct 30 23:19:01.355787 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:01.355705 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b81a7a-5d9fb6cfb7-6wttp" Oct 30 23:19:01.417998 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:01.417968 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec586e23-efdd-4fd1-b54c-3c11b5d90ae1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-3c9613-75d9965b8c-g924m\" (UID: \"ec586e23-efdd-4fd1-b54c-3c11b5d90ae1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3c9613-75d9965b8c-g924m" Oct 30 23:19:01.418207 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:01.418183 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srf55\" (UniqueName: \"kubernetes.io/projected/ec586e23-efdd-4fd1-b54c-3c11b5d90ae1-kube-api-access-srf55\") pod \"payload-pre-processing-e2e-worker-w2-3c9613-75d9965b8c-g924m\" (UID: \"ec586e23-efdd-4fd1-b54c-3c11b5d90ae1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3c9613-75d9965b8c-g924m" Oct 30 23:19:01.418785 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:01.418765 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec586e23-efdd-4fd1-b54c-3c11b5d90ae1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-3c9613-75d9965b8c-g924m\" (UID: \"ec586e23-efdd-4fd1-b54c-3c11b5d90ae1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3c9613-75d9965b8c-g924m" Oct 30 23:19:01.479219 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:01.479153 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srf55\" (UniqueName: \"kubernetes.io/projected/ec586e23-efdd-4fd1-b54c-3c11b5d90ae1-kube-api-access-srf55\") pod \"payload-pre-processing-e2e-worker-w2-3c9613-75d9965b8c-g924m\" (UID: \"ec586e23-efdd-4fd1-b54c-3c11b5d90ae1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3c9613-75d9965b8c-g924m" Oct 30 23:19:01.520125 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:01.520083 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b81a7a-5d9fb6cfb7-6wttp" podStartSLOduration=2.6452249180000003 podStartE2EDuration="3.520068839s" totalImagesPullingTime="874.843921ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:18:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:01.519432861 +0000 UTC m=+760.663263067" watchObservedRunningTime="2026-10-30 23:19:01.520068839 +0000 UTC m=+760.663899055" Oct 30 23:19:01.601997 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:01.601969 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3c9613-75d9965b8c-g924m" Oct 30 23:19:01.636750 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:01.636704 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-b81a7a-78746cfd9c-ql62c" podStartSLOduration=2.782722805 podStartE2EDuration="3.636688714s" totalImagesPullingTime="853.965909ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:18:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:01.633970072 +0000 UTC m=+760.777800310" watchObservedRunningTime="2026-10-30 23:19:01.636688714 +0000 UTC m=+760.780518950" Oct 30 23:19:01.974448 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:01.974420 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3c9613-75d9965b8c-g924m"] Oct 30 23:19:01.975471 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:19:01.975447 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec586e23_efdd_4fd1_b54c_3c11b5d90ae1.slice/crio-cdd1f80ec91ce3241ef250d55a9d3accbcd2ef5803d3fa34cbff6928580ffa6e WatchSource:0}: Error finding container cdd1f80ec91ce3241ef250d55a9d3accbcd2ef5803d3fa34cbff6928580ffa6e: Status 404 returned error can't find the container with id cdd1f80ec91ce3241ef250d55a9d3accbcd2ef5803d3fa34cbff6928580ffa6e Oct 30 23:19:02.362164 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:02.362130 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3c9613-75d9965b8c-g924m" event={"ID":"ec586e23-efdd-4fd1-b54c-3c11b5d90ae1","Type":"ContainerStarted","Data":"4c7cd9678ca6c652205f11c659e992ca3e0cf0915f2977379470fed7afb534be"} Oct 30 23:19:02.362406 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:02.362189 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3c9613-75d9965b8c-g924m" event={"ID":"ec586e23-efdd-4fd1-b54c-3c11b5d90ae1","Type":"ContainerStarted","Data":"cdd1f80ec91ce3241ef250d55a9d3accbcd2ef5803d3fa34cbff6928580ffa6e"} Oct 30 23:19:02.483950 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:02.483903 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3c9613-75d9965b8c-g924m" podStartSLOduration=1.483890985 podStartE2EDuration="1.483890985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:02.482178725 +0000 UTC m=+761.626008952" watchObservedRunningTime="2026-10-30 23:19:02.483890985 +0000 UTC m=+761.627721230" Oct 30 23:19:03.367454 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:03.367424 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3c9613-75d9965b8c-g924m" Oct 30 23:19:05.962078 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:05.962029 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:06.201224 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:06.201186 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:07.315999 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:07.315956 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:19:10.716864 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:10.716831 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6"] Oct 30 23:19:10.720438 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:10.720420 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6" Oct 30 23:19:10.786084 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:10.786009 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f90492f3-9a0a-447d-8a29-6a91e026a5b1-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6\" (UID: \"f90492f3-9a0a-447d-8a29-6a91e026a5b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6" Oct 30 23:19:10.786227 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:10.786123 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f90492f3-9a0a-447d-8a29-6a91e026a5b1-maas-api-tls\") pod \"maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6\" (UID: \"f90492f3-9a0a-447d-8a29-6a91e026a5b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6" Oct 30 23:19:10.786227 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:10.786161 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f90492f3-9a0a-447d-8a29-6a91e026a5b1-metrics-certs\") pod \"maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6\" (UID: \"f90492f3-9a0a-447d-8a29-6a91e026a5b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6" Oct 30 23:19:10.786356 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:10.786252 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rljbz\" (UniqueName: \"kubernetes.io/projected/f90492f3-9a0a-447d-8a29-6a91e026a5b1-kube-api-access-rljbz\") pod \"maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6\" (UID: \"f90492f3-9a0a-447d-8a29-6a91e026a5b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6" Oct 30 23:19:10.823428 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:10.823396 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-989a4f-serving-cert\"" Oct 30 23:19:10.823553 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:10.823500 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s5prt\"" Oct 30 23:19:10.823553 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:10.823540 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 23:19:10.825977 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:10.825956 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6"] Oct 30 23:19:10.886784 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:10.886757 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f90492f3-9a0a-447d-8a29-6a91e026a5b1-metrics-certs\") pod \"maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6\" (UID: \"f90492f3-9a0a-447d-8a29-6a91e026a5b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6" Oct 30 23:19:10.886784 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:10.886820 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rljbz\" (UniqueName: \"kubernetes.io/projected/f90492f3-9a0a-447d-8a29-6a91e026a5b1-kube-api-access-rljbz\") pod \"maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6\" (UID: \"f90492f3-9a0a-447d-8a29-6a91e026a5b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6" Oct 30 23:19:10.887126 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:10.886918 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f90492f3-9a0a-447d-8a29-6a91e026a5b1-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6\" (UID: \"f90492f3-9a0a-447d-8a29-6a91e026a5b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6" Oct 30 23:19:10.887126 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:10.886957 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f90492f3-9a0a-447d-8a29-6a91e026a5b1-maas-api-tls\") pod \"maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6\" (UID: \"f90492f3-9a0a-447d-8a29-6a91e026a5b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6" Oct 30 23:19:10.887239 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:19:10.887128 4952 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-989a4f-serving-cert: secret "maas-api-e2e-worker-w4-989a4f-serving-cert" not found Oct 30 23:19:10.887239 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:19:10.887197 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f90492f3-9a0a-447d-8a29-6a91e026a5b1-maas-api-tls podName:f90492f3-9a0a-447d-8a29-6a91e026a5b1 nodeName:}" failed. No retries permitted until 2026-10-30 23:19:11.387176349 +0000 UTC m=+770.531006545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f90492f3-9a0a-447d-8a29-6a91e026a5b1-maas-api-tls") pod "maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6" (UID: "f90492f3-9a0a-447d-8a29-6a91e026a5b1") : secret "maas-api-e2e-worker-w4-989a4f-serving-cert" not found Oct 30 23:19:10.888534 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:10.888505 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f90492f3-9a0a-447d-8a29-6a91e026a5b1-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6\" (UID: \"f90492f3-9a0a-447d-8a29-6a91e026a5b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6" Oct 30 23:19:10.890149 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:10.890123 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f90492f3-9a0a-447d-8a29-6a91e026a5b1-metrics-certs\") pod \"maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6\" (UID: \"f90492f3-9a0a-447d-8a29-6a91e026a5b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6" Oct 30 23:19:10.943842 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:10.943812 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rljbz\" (UniqueName: \"kubernetes.io/projected/f90492f3-9a0a-447d-8a29-6a91e026a5b1-kube-api-access-rljbz\") pod \"maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6\" (UID: \"f90492f3-9a0a-447d-8a29-6a91e026a5b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6" Oct 30 23:19:10.996081 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:10.996002 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:11.214080 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:11.214030 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:11.359592 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:11.359568 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aadaa6-85599b78b7-mz9sj" Oct 30 23:19:11.392030 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:11.392005 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f90492f3-9a0a-447d-8a29-6a91e026a5b1-maas-api-tls\") pod \"maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6\" (UID: \"f90492f3-9a0a-447d-8a29-6a91e026a5b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6" Oct 30 23:19:11.394696 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:11.394670 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f90492f3-9a0a-447d-8a29-6a91e026a5b1-maas-api-tls\") pod \"maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6\" (UID: \"f90492f3-9a0a-447d-8a29-6a91e026a5b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6" Oct 30 23:19:11.570718 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:11.570689 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:11.634011 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:11.633943 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6" Oct 30 23:19:12.056560 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:12.056533 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6"] Oct 30 23:19:12.057657 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:19:12.057624 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf90492f3_9a0a_447d_8a29_6a91e026a5b1.slice/crio-92c0f3fb2e15a01d955adabea84c55192779d88c3b74a23ca53eff16b9d38268 WatchSource:0}: Error finding container 92c0f3fb2e15a01d955adabea84c55192779d88c3b74a23ca53eff16b9d38268: Status 404 returned error can't find the container with id 92c0f3fb2e15a01d955adabea84c55192779d88c3b74a23ca53eff16b9d38268 Oct 30 23:19:12.365746 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:12.365712 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b81a7a-5d9fb6cfb7-6wttp" Oct 30 23:19:12.366124 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:12.366104 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-b81a7a-78746cfd9c-ql62c" Oct 30 23:19:12.421304 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:12.421271 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6" event={"ID":"f90492f3-9a0a-447d-8a29-6a91e026a5b1","Type":"ContainerStarted","Data":"92c0f3fb2e15a01d955adabea84c55192779d88c3b74a23ca53eff16b9d38268"} Oct 30 23:19:14.375142 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:14.375117 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3c9613-75d9965b8c-g924m" Oct 30 23:19:14.433289 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:14.433259 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6" event={"ID":"f90492f3-9a0a-447d-8a29-6a91e026a5b1","Type":"ContainerStarted","Data":"2021a6717a0e0e7cf748a5eca59cbe799e8e372a1a472b3e54d81c247cebb4d5"} Oct 30 23:19:14.433422 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:14.433347 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6" Oct 30 23:19:14.606202 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:14.606150 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6" podStartSLOduration=2.383870199 podStartE2EDuration="4.606135814s" totalImagesPullingTime="2.222265615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:14.603993214 +0000 UTC m=+773.747823420" watchObservedRunningTime="2026-10-30 23:19:14.606135814 +0000 UTC m=+773.749966019" Oct 30 23:19:14.795261 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:14.795225 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:16.073117 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:16.073086 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:18.722915 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:18.722883 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:19:20.445361 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:20.445330 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6" Oct 30 23:19:20.625251 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:20.625216 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk"] Oct 30 23:19:20.630862 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:20.630838 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk" Oct 30 23:19:20.674195 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:20.674173 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82q2d\" (UniqueName: \"kubernetes.io/projected/dd4ad911-4b05-4bf3-82c4-9c368ce50137-kube-api-access-82q2d\") pod \"maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk\" (UID: \"dd4ad911-4b05-4bf3-82c4-9c368ce50137\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk" Oct 30 23:19:20.674195 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:20.674201 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd4ad911-4b05-4bf3-82c4-9c368ce50137-maas-api-tls\") pod \"maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk\" (UID: \"dd4ad911-4b05-4bf3-82c4-9c368ce50137\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk" Oct 30 23:19:20.674384 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:20.674300 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd4ad911-4b05-4bf3-82c4-9c368ce50137-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk\" (UID: \"dd4ad911-4b05-4bf3-82c4-9c368ce50137\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk" Oct 30 23:19:20.674423 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:20.674399 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd4ad911-4b05-4bf3-82c4-9c368ce50137-metrics-certs\") pod \"maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk\" (UID: \"dd4ad911-4b05-4bf3-82c4-9c368ce50137\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk" Oct 30 23:19:20.683306 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:20.683277 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-2cfb80-serving-cert\"" Oct 30 23:19:20.686159 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:20.686128 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk"] Oct 30 23:19:20.775316 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:20.775245 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82q2d\" (UniqueName: \"kubernetes.io/projected/dd4ad911-4b05-4bf3-82c4-9c368ce50137-kube-api-access-82q2d\") pod \"maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk\" (UID: \"dd4ad911-4b05-4bf3-82c4-9c368ce50137\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk" Oct 30 23:19:20.775316 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:20.775285 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd4ad911-4b05-4bf3-82c4-9c368ce50137-maas-api-tls\") pod \"maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk\" (UID: \"dd4ad911-4b05-4bf3-82c4-9c368ce50137\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk" Oct 30 23:19:20.775478 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:20.775357 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd4ad911-4b05-4bf3-82c4-9c368ce50137-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk\" (UID: \"dd4ad911-4b05-4bf3-82c4-9c368ce50137\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk" Oct 30 23:19:20.775478 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:20.775415 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd4ad911-4b05-4bf3-82c4-9c368ce50137-metrics-certs\") pod \"maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk\" (UID: \"dd4ad911-4b05-4bf3-82c4-9c368ce50137\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk" Oct 30 23:19:20.775478 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:19:20.775445 4952 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-2cfb80-serving-cert: secret "maas-api-e2e-worker-w3-2cfb80-serving-cert" not found Oct 30 23:19:20.775602 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:19:20.775510 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd4ad911-4b05-4bf3-82c4-9c368ce50137-maas-api-tls podName:dd4ad911-4b05-4bf3-82c4-9c368ce50137 nodeName:}" failed. No retries permitted until 2026-10-30 23:19:21.275489543 +0000 UTC m=+780.419319727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dd4ad911-4b05-4bf3-82c4-9c368ce50137-maas-api-tls") pod "maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk" (UID: "dd4ad911-4b05-4bf3-82c4-9c368ce50137") : secret "maas-api-e2e-worker-w3-2cfb80-serving-cert" not found Oct 30 23:19:20.776298 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:20.776274 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd4ad911-4b05-4bf3-82c4-9c368ce50137-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk\" (UID: \"dd4ad911-4b05-4bf3-82c4-9c368ce50137\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk" Oct 30 23:19:20.778142 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:20.778121 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd4ad911-4b05-4bf3-82c4-9c368ce50137-metrics-certs\") pod \"maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk\" (UID: \"dd4ad911-4b05-4bf3-82c4-9c368ce50137\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk" Oct 30 23:19:20.832522 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:20.832498 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82q2d\" (UniqueName: \"kubernetes.io/projected/dd4ad911-4b05-4bf3-82c4-9c368ce50137-kube-api-access-82q2d\") pod \"maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk\" (UID: \"dd4ad911-4b05-4bf3-82c4-9c368ce50137\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk" Oct 30 23:19:20.901497 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:20.901463 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:21.127237 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:21.127205 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:21.140782 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:19:21.140753 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:19:21.279054 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:21.279008 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd4ad911-4b05-4bf3-82c4-9c368ce50137-maas-api-tls\") pod \"maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk\" (UID: \"dd4ad911-4b05-4bf3-82c4-9c368ce50137\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk" Oct 30 23:19:21.281913 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:21.281881 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd4ad911-4b05-4bf3-82c4-9c368ce50137-maas-api-tls\") pod \"maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk\" (UID: \"dd4ad911-4b05-4bf3-82c4-9c368ce50137\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk" Oct 30 23:19:21.545668 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:21.545602 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk" Oct 30 23:19:21.925575 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:21.925548 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk"] Oct 30 23:19:21.926266 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:19:21.926229 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd4ad911_4b05_4bf3_82c4_9c368ce50137.slice/crio-2f480e2053565f8f31e141d50ea53e7804f2dd938f15260eb5ef2e5b056b8b0e WatchSource:0}: Error finding container 2f480e2053565f8f31e141d50ea53e7804f2dd938f15260eb5ef2e5b056b8b0e: Status 404 returned error can't find the container with id 2f480e2053565f8f31e141d50ea53e7804f2dd938f15260eb5ef2e5b056b8b0e Oct 30 23:19:22.479811 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:22.479777 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk" event={"ID":"dd4ad911-4b05-4bf3-82c4-9c368ce50137","Type":"ContainerStarted","Data":"2f480e2053565f8f31e141d50ea53e7804f2dd938f15260eb5ef2e5b056b8b0e"} Oct 30 23:19:23.488743 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:23.488706 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk" event={"ID":"dd4ad911-4b05-4bf3-82c4-9c368ce50137","Type":"ContainerStarted","Data":"4ed8bfbcf1cd8dc3f73b752465e6002016defb78b15728e9a5b666be4b2d559f"} Oct 30 23:19:23.489172 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:23.488913 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk" Oct 30 23:19:23.608332 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:23.608286 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk" podStartSLOduration=2.808515232 podStartE2EDuration="3.608273807s" totalImagesPullingTime="799.758575ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:23.607507766 +0000 UTC m=+782.751337991" watchObservedRunningTime="2026-10-30 23:19:23.608273807 +0000 UTC m=+782.752104012" Oct 30 23:19:25.804199 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:25.804155 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:26.010513 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:26.010459 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:26.229266 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:26.229227 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:29.500069 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:29.500025 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk" Oct 30 23:19:30.397436 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:30.397402 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:19:30.896580 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:30.896548 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:35.855854 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:35.855822 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:38.570682 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:38.570647 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:39.134223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:39.134188 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:39.499553 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:39.499484 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:39.817415 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:39.817345 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld"] Oct 30 23:19:39.821122 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:39.821105 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" Oct 30 23:19:39.937934 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:39.937898 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htxb8\" (UniqueName: \"kubernetes.io/projected/3a7312d5-8075-4543-a074-1f176cf36911-kube-api-access-htxb8\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" Oct 30 23:19:39.938118 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:39.937946 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a7312d5-8075-4543-a074-1f176cf36911-istio-envoy\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" Oct 30 23:19:39.938118 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:39.937963 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a7312d5-8075-4543-a074-1f176cf36911-istio-data\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" Oct 30 23:19:39.938118 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:39.938103 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7312d5-8075-4543-a074-1f176cf36911-credential-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" Oct 30 23:19:39.938317 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:39.938134 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a7312d5-8075-4543-a074-1f176cf36911-istiod-ca-cert\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" Oct 30 23:19:39.938317 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:39.938160 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a7312d5-8075-4543-a074-1f176cf36911-istio-podinfo\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" Oct 30 23:19:39.938317 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:39.938258 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7312d5-8075-4543-a074-1f176cf36911-workload-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" Oct 30 23:19:39.938317 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:39.938294 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a7312d5-8075-4543-a074-1f176cf36911-workload-certs\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" Oct 30 23:19:39.938317 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:39.938312 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a7312d5-8075-4543-a074-1f176cf36911-istio-token\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" Oct 30 23:19:39.991694 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:39.991668 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-2cfb80-openshift-default-dockercfg-zk28c\"" Oct 30 23:19:40.031289 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:40.031263 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld"] Oct 30 23:19:40.039346 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:40.039322 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7312d5-8075-4543-a074-1f176cf36911-credential-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" Oct 30 23:19:40.039463 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:40.039355 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a7312d5-8075-4543-a074-1f176cf36911-istiod-ca-cert\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" Oct 30 23:19:40.039463 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:40.039386 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a7312d5-8075-4543-a074-1f176cf36911-istio-podinfo\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" Oct 30 23:19:40.039463 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:40.039407 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7312d5-8075-4543-a074-1f176cf36911-workload-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" Oct 30 23:19:40.039463 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:40.039455 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a7312d5-8075-4543-a074-1f176cf36911-workload-certs\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" Oct 30 23:19:40.039702 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:40.039676 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a7312d5-8075-4543-a074-1f176cf36911-istio-token\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" Oct 30 23:19:40.039763 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:40.039748 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htxb8\" (UniqueName: \"kubernetes.io/projected/3a7312d5-8075-4543-a074-1f176cf36911-kube-api-access-htxb8\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" Oct 30 23:19:40.039846 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:40.039829 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a7312d5-8075-4543-a074-1f176cf36911-istio-envoy\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" Oct 30 23:19:40.039905 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:40.039858 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a7312d5-8075-4543-a074-1f176cf36911-istio-data\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" Oct 30 23:19:40.039968 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:40.039901 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7312d5-8075-4543-a074-1f176cf36911-workload-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" Oct 30 23:19:40.040178 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:40.040157 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a7312d5-8075-4543-a074-1f176cf36911-istiod-ca-cert\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" Oct 30 23:19:40.040247 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:40.040223 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a7312d5-8075-4543-a074-1f176cf36911-istio-data\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" Oct 30 23:19:40.040474 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:40.040453 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a7312d5-8075-4543-a074-1f176cf36911-workload-certs\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" Oct 30 23:19:40.040708 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:40.040510 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7312d5-8075-4543-a074-1f176cf36911-credential-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" Oct 30 23:19:40.042235 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:40.042214 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a7312d5-8075-4543-a074-1f176cf36911-istio-envoy\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" Oct 30 23:19:40.042685 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:40.042669 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a7312d5-8075-4543-a074-1f176cf36911-istio-podinfo\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" Oct 30 23:19:40.099709 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:40.099684 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a7312d5-8075-4543-a074-1f176cf36911-istio-token\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" Oct 30 23:19:40.099917 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:40.099893 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htxb8\" (UniqueName: \"kubernetes.io/projected/3a7312d5-8075-4543-a074-1f176cf36911-kube-api-access-htxb8\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" Oct 30 23:19:40.134022 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:40.134000 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" Oct 30 23:19:40.540749 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:40.540668 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:19:40.565411 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:40.565385 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld"] Oct 30 23:19:40.567070 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:19:40.567006 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a7312d5_8075_4543_a074_1f176cf36911.slice/crio-afa27661dd85d31ff5e120a73567f38653fc45fac73272ea209b8a4d65a1847d WatchSource:0}: Error finding container afa27661dd85d31ff5e120a73567f38653fc45fac73272ea209b8a4d65a1847d: Status 404 returned error can't find the container with id afa27661dd85d31ff5e120a73567f38653fc45fac73272ea209b8a4d65a1847d Oct 30 23:19:40.569832 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:40.569801 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:19:40.569933 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:40.569882 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:19:40.569933 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:40.569925 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:19:40.591798 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:40.591771 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" event={"ID":"3a7312d5-8075-4543-a074-1f176cf36911","Type":"ContainerStarted","Data":"afa27661dd85d31ff5e120a73567f38653fc45fac73272ea209b8a4d65a1847d"} Oct 30 23:19:41.220365 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:41.220332 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:41.598742 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:41.598702 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" event={"ID":"3a7312d5-8075-4543-a074-1f176cf36911","Type":"ContainerStarted","Data":"49caa245a97c277c2ac3e396ce634547f91a811b7ea94d2039bded1c963c884b"} Oct 30 23:19:41.816350 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:41.816292 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" podStartSLOduration=2.816276566 podStartE2EDuration="2.816276566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:41.764342062 +0000 UTC m=+800.908172268" watchObservedRunningTime="2026-10-30 23:19:41.816276566 +0000 UTC m=+800.960106833" Oct 30 23:19:42.090060 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:42.088745 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw"] Oct 30 23:19:42.093552 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:42.093531 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" Oct 30 23:19:42.134877 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:42.134848 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" Oct 30 23:19:42.136674 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:42.136643 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 23:19:42.136865 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:42.136687 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" podUID="3a7312d5-8075-4543-a074-1f176cf36911" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 23:19:42.148920 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:42.148874 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw"] Oct 30 23:19:42.149554 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:42.149523 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-057303\"/\"kube-root-ca.crt\"" Oct 30 23:19:42.149659 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:42.149635 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-057303\"/\"openshift-service-ca.crt\"" Oct 30 23:19:42.149715 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:42.149639 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-057303\"/\"default-dockercfg-csbqq\"" Oct 30 23:19:42.163310 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:42.163288 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86ec338e-ad13-4029-a1cc-88650c6a27ca-home\") pod \"simulator-w0-057303-kserve-66b7c9b57f-wt5qw\" (UID: \"86ec338e-ad13-4029-a1cc-88650c6a27ca\") " pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" Oct 30 23:19:42.163399 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:42.163327 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2phtn\" (UniqueName: \"kubernetes.io/projected/86ec338e-ad13-4029-a1cc-88650c6a27ca-kube-api-access-2phtn\") pod \"simulator-w0-057303-kserve-66b7c9b57f-wt5qw\" (UID: \"86ec338e-ad13-4029-a1cc-88650c6a27ca\") " pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" Oct 30 23:19:42.163399 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:42.163362 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86ec338e-ad13-4029-a1cc-88650c6a27ca-tmp-dir\") pod \"simulator-w0-057303-kserve-66b7c9b57f-wt5qw\" (UID: \"86ec338e-ad13-4029-a1cc-88650c6a27ca\") " pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" Oct 30 23:19:42.163498 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:42.163480 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86ec338e-ad13-4029-a1cc-88650c6a27ca-model-cache\") pod \"simulator-w0-057303-kserve-66b7c9b57f-wt5qw\" (UID: \"86ec338e-ad13-4029-a1cc-88650c6a27ca\") " pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" Oct 30 23:19:42.163541 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:42.163528 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86ec338e-ad13-4029-a1cc-88650c6a27ca-tls-certs\") pod \"simulator-w0-057303-kserve-66b7c9b57f-wt5qw\" (UID: \"86ec338e-ad13-4029-a1cc-88650c6a27ca\") " pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" Oct 30 23:19:42.163626 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:42.163613 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86ec338e-ad13-4029-a1cc-88650c6a27ca-dshm\") pod \"simulator-w0-057303-kserve-66b7c9b57f-wt5qw\" (UID: \"86ec338e-ad13-4029-a1cc-88650c6a27ca\") " pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" Oct 30 23:19:42.190249 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:42.190221 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-057303\"/\"simulator-w0-057303-kserve-self-signed-certs\"" Oct 30 23:19:42.264420 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:42.264391 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86ec338e-ad13-4029-a1cc-88650c6a27ca-tls-certs\") pod \"simulator-w0-057303-kserve-66b7c9b57f-wt5qw\" (UID: \"86ec338e-ad13-4029-a1cc-88650c6a27ca\") " pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" Oct 30 23:19:42.264840 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:42.264445 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86ec338e-ad13-4029-a1cc-88650c6a27ca-dshm\") pod \"simulator-w0-057303-kserve-66b7c9b57f-wt5qw\" (UID: \"86ec338e-ad13-4029-a1cc-88650c6a27ca\") " pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" Oct 30 23:19:42.264840 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:42.264496 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86ec338e-ad13-4029-a1cc-88650c6a27ca-home\") pod \"simulator-w0-057303-kserve-66b7c9b57f-wt5qw\" (UID: \"86ec338e-ad13-4029-a1cc-88650c6a27ca\") " pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" Oct 30 23:19:42.264840 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:42.264547 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2phtn\" (UniqueName: \"kubernetes.io/projected/86ec338e-ad13-4029-a1cc-88650c6a27ca-kube-api-access-2phtn\") pod \"simulator-w0-057303-kserve-66b7c9b57f-wt5qw\" (UID: \"86ec338e-ad13-4029-a1cc-88650c6a27ca\") " pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" Oct 30 23:19:42.264840 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:42.264600 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86ec338e-ad13-4029-a1cc-88650c6a27ca-tmp-dir\") pod \"simulator-w0-057303-kserve-66b7c9b57f-wt5qw\" (UID: \"86ec338e-ad13-4029-a1cc-88650c6a27ca\") " pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" Oct 30 23:19:42.264840 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:42.264626 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86ec338e-ad13-4029-a1cc-88650c6a27ca-model-cache\") pod \"simulator-w0-057303-kserve-66b7c9b57f-wt5qw\" (UID: \"86ec338e-ad13-4029-a1cc-88650c6a27ca\") " pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" Oct 30 23:19:42.265142 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:42.265105 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86ec338e-ad13-4029-a1cc-88650c6a27ca-model-cache\") pod \"simulator-w0-057303-kserve-66b7c9b57f-wt5qw\" (UID: \"86ec338e-ad13-4029-a1cc-88650c6a27ca\") " pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" Oct 30 23:19:42.265209 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:42.265179 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86ec338e-ad13-4029-a1cc-88650c6a27ca-tmp-dir\") pod \"simulator-w0-057303-kserve-66b7c9b57f-wt5qw\" (UID: \"86ec338e-ad13-4029-a1cc-88650c6a27ca\") " pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" Oct 30 23:19:42.285099 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:42.265106 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86ec338e-ad13-4029-a1cc-88650c6a27ca-home\") pod \"simulator-w0-057303-kserve-66b7c9b57f-wt5qw\" (UID: \"86ec338e-ad13-4029-a1cc-88650c6a27ca\") " pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" Oct 30 23:19:42.285099 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:42.275681 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86ec338e-ad13-4029-a1cc-88650c6a27ca-tls-certs\") pod \"simulator-w0-057303-kserve-66b7c9b57f-wt5qw\" (UID: \"86ec338e-ad13-4029-a1cc-88650c6a27ca\") " pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" Oct 30 23:19:42.285099 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:42.281340 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:42.296630 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:42.296604 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86ec338e-ad13-4029-a1cc-88650c6a27ca-dshm\") pod \"simulator-w0-057303-kserve-66b7c9b57f-wt5qw\" (UID: \"86ec338e-ad13-4029-a1cc-88650c6a27ca\") " pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" Oct 30 23:19:42.322641 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:42.322618 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2phtn\" (UniqueName: \"kubernetes.io/projected/86ec338e-ad13-4029-a1cc-88650c6a27ca-kube-api-access-2phtn\") pod \"simulator-w0-057303-kserve-66b7c9b57f-wt5qw\" (UID: \"86ec338e-ad13-4029-a1cc-88650c6a27ca\") " pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" Oct 30 23:19:42.407451 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:42.407332 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" Oct 30 23:19:42.792763 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:42.792736 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw"] Oct 30 23:19:42.795564 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:19:42.795535 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86ec338e_ad13_4029_a1cc_88650c6a27ca.slice/crio-b7c6ec0666ab4ab3ad12c8971d0b77c96e4a16a2e0c3022fd524bf4200e1829c WatchSource:0}: Error finding container b7c6ec0666ab4ab3ad12c8971d0b77c96e4a16a2e0c3022fd524bf4200e1829c: Status 404 returned error can't find the container with id b7c6ec0666ab4ab3ad12c8971d0b77c96e4a16a2e0c3022fd524bf4200e1829c Oct 30 23:19:43.134416 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:43.134380 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 23:19:43.134582 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:43.134436 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" podUID="3a7312d5-8075-4543-a074-1f176cf36911" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 23:19:43.630094 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:43.616779 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" event={"ID":"86ec338e-ad13-4029-a1cc-88650c6a27ca","Type":"ContainerStarted","Data":"b7c6ec0666ab4ab3ad12c8971d0b77c96e4a16a2e0c3022fd524bf4200e1829c"} Oct 30 23:19:43.791005 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:43.790957 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:44.135266 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:44.135231 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 23:19:44.135448 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:44.135293 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" podUID="3a7312d5-8075-4543-a074-1f176cf36911" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 23:19:44.396112 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:44.395828 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:44.627131 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:44.627021 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:45.134704 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:45.134665 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 23:19:45.135209 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:45.134722 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" podUID="3a7312d5-8075-4543-a074-1f176cf36911" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 23:19:45.633544 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:45.633509 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" event={"ID":"86ec338e-ad13-4029-a1cc-88650c6a27ca","Type":"ContainerStarted","Data":"4093eb89cc76ec4aa91987653e2497364397fbf33fe1283aef0cf97ecb2d202d"} Oct 30 23:19:46.134395 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:46.134357 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 23:19:46.134563 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:46.134419 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" podUID="3a7312d5-8075-4543-a074-1f176cf36911" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 23:19:46.408550 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:46.408463 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" Oct 30 23:19:46.633696 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:46.633654 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:46.865229 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:46.865200 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:47.138650 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:47.138579 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" Oct 30 23:19:47.139100 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:47.139078 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" Oct 30 23:19:47.139848 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:47.139830 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" Oct 30 23:19:47.306632 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:47.306580 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" podStartSLOduration=3.394233984 podStartE2EDuration="5.306565351s" totalImagesPullingTime="1.912331367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:45.758588543 +0000 UTC m=+804.902418765" watchObservedRunningTime="2026-10-30 23:19:47.306565351 +0000 UTC m=+806.450395557" Oct 30 23:19:48.034280 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:48.034249 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:48.233098 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:48.233069 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:50.950055 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:50.950016 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:51.141673 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:19:51.141637 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:19:51.542302 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:51.542268 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:51.826088 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:51.825988 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:52.108677 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:52.108644 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:52.407590 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:52.407507 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" Oct 30 23:19:52.420472 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:52.420441 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" Oct 30 23:19:52.688884 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:52.688811 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" Oct 30 23:19:54.789926 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:54.789896 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:19:54.962216 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:54.962185 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:55.532710 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:55.532677 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:55.763919 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:55.763889 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:57.099605 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:57.099571 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:58.129898 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:58.129862 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:58.784875 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:58.784842 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:19:59.068454 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:19:59.068365 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:20:02.630851 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:02.630819 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:20:06.931249 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:06.931218 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq"] Oct 30 23:20:06.934980 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:06.934962 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" Oct 30 23:20:07.020379 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:07.020351 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/145e7c01-efec-46e6-a748-56b39de76f83-tmp-dir\") pod \"simulator-w6-aadaa6-kserve-9795784cf-skbvq\" (UID: \"145e7c01-efec-46e6-a748-56b39de76f83\") " pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" Oct 30 23:20:07.020525 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:07.020408 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/145e7c01-efec-46e6-a748-56b39de76f83-dshm\") pod \"simulator-w6-aadaa6-kserve-9795784cf-skbvq\" (UID: \"145e7c01-efec-46e6-a748-56b39de76f83\") " pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" Oct 30 23:20:07.020525 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:07.020496 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/145e7c01-efec-46e6-a748-56b39de76f83-model-cache\") pod \"simulator-w6-aadaa6-kserve-9795784cf-skbvq\" (UID: \"145e7c01-efec-46e6-a748-56b39de76f83\") " pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" Oct 30 23:20:07.020601 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:07.020568 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/145e7c01-efec-46e6-a748-56b39de76f83-tls-certs\") pod \"simulator-w6-aadaa6-kserve-9795784cf-skbvq\" (UID: \"145e7c01-efec-46e6-a748-56b39de76f83\") " pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" Oct 30 23:20:07.020636 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:07.020622 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/145e7c01-efec-46e6-a748-56b39de76f83-home\") pod \"simulator-w6-aadaa6-kserve-9795784cf-skbvq\" (UID: \"145e7c01-efec-46e6-a748-56b39de76f83\") " pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" Oct 30 23:20:07.020689 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:07.020673 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmtvh\" (UniqueName: \"kubernetes.io/projected/145e7c01-efec-46e6-a748-56b39de76f83-kube-api-access-nmtvh\") pod \"simulator-w6-aadaa6-kserve-9795784cf-skbvq\" (UID: \"145e7c01-efec-46e6-a748-56b39de76f83\") " pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" Oct 30 23:20:07.037528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:07.037503 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-aadaa6\"/\"openshift-service-ca.crt\"" Oct 30 23:20:07.037528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:07.037504 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-aadaa6\"/\"default-dockercfg-d4g98\"" Oct 30 23:20:07.037718 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:07.037569 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-aadaa6\"/\"kube-root-ca.crt\"" Oct 30 23:20:07.037718 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:07.037647 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-aadaa6\"/\"simulator-w6-aadaa6-kserve-self-signed-certs\"" Oct 30 23:20:07.040353 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:07.040335 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq"] Oct 30 23:20:07.111406 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:07.111384 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:20:07.121611 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:07.121583 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmtvh\" (UniqueName: \"kubernetes.io/projected/145e7c01-efec-46e6-a748-56b39de76f83-kube-api-access-nmtvh\") pod \"simulator-w6-aadaa6-kserve-9795784cf-skbvq\" (UID: \"145e7c01-efec-46e6-a748-56b39de76f83\") " pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" Oct 30 23:20:07.121741 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:07.121636 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/145e7c01-efec-46e6-a748-56b39de76f83-tmp-dir\") pod \"simulator-w6-aadaa6-kserve-9795784cf-skbvq\" (UID: \"145e7c01-efec-46e6-a748-56b39de76f83\") " pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" Oct 30 23:20:07.121741 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:07.121708 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/145e7c01-efec-46e6-a748-56b39de76f83-dshm\") pod \"simulator-w6-aadaa6-kserve-9795784cf-skbvq\" (UID: \"145e7c01-efec-46e6-a748-56b39de76f83\") " pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" Oct 30 23:20:07.121856 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:07.121744 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/145e7c01-efec-46e6-a748-56b39de76f83-model-cache\") pod \"simulator-w6-aadaa6-kserve-9795784cf-skbvq\" (UID: \"145e7c01-efec-46e6-a748-56b39de76f83\") " pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" Oct 30 23:20:07.121856 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:07.121780 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/145e7c01-efec-46e6-a748-56b39de76f83-tls-certs\") pod \"simulator-w6-aadaa6-kserve-9795784cf-skbvq\" (UID: \"145e7c01-efec-46e6-a748-56b39de76f83\") " pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" Oct 30 23:20:07.121856 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:07.121828 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/145e7c01-efec-46e6-a748-56b39de76f83-home\") pod \"simulator-w6-aadaa6-kserve-9795784cf-skbvq\" (UID: \"145e7c01-efec-46e6-a748-56b39de76f83\") " pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" Oct 30 23:20:07.122153 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:07.122127 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/145e7c01-efec-46e6-a748-56b39de76f83-tmp-dir\") pod \"simulator-w6-aadaa6-kserve-9795784cf-skbvq\" (UID: \"145e7c01-efec-46e6-a748-56b39de76f83\") " pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" Oct 30 23:20:07.122324 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:07.122298 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/145e7c01-efec-46e6-a748-56b39de76f83-home\") pod \"simulator-w6-aadaa6-kserve-9795784cf-skbvq\" (UID: \"145e7c01-efec-46e6-a748-56b39de76f83\") " pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" Oct 30 23:20:07.122454 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:07.122251 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/145e7c01-efec-46e6-a748-56b39de76f83-model-cache\") pod \"simulator-w6-aadaa6-kserve-9795784cf-skbvq\" (UID: \"145e7c01-efec-46e6-a748-56b39de76f83\") " pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" Oct 30 23:20:07.124392 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:07.124364 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/145e7c01-efec-46e6-a748-56b39de76f83-dshm\") pod \"simulator-w6-aadaa6-kserve-9795784cf-skbvq\" (UID: \"145e7c01-efec-46e6-a748-56b39de76f83\") " pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" Oct 30 23:20:07.125180 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:07.125157 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/145e7c01-efec-46e6-a748-56b39de76f83-tls-certs\") pod \"simulator-w6-aadaa6-kserve-9795784cf-skbvq\" (UID: \"145e7c01-efec-46e6-a748-56b39de76f83\") " pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" Oct 30 23:20:07.178884 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:07.178852 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmtvh\" (UniqueName: \"kubernetes.io/projected/145e7c01-efec-46e6-a748-56b39de76f83-kube-api-access-nmtvh\") pod \"simulator-w6-aadaa6-kserve-9795784cf-skbvq\" (UID: \"145e7c01-efec-46e6-a748-56b39de76f83\") " pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" Oct 30 23:20:07.246941 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:07.246843 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" Oct 30 23:20:07.646789 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:07.646764 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq"] Oct 30 23:20:07.647664 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:20:07.647644 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod145e7c01_efec_46e6_a748_56b39de76f83.slice/crio-85c2798843abd3594b37119e9188654f94f1bf939b4a34862fa62d25d63d0b37 WatchSource:0}: Error finding container 85c2798843abd3594b37119e9188654f94f1bf939b4a34862fa62d25d63d0b37: Status 404 returned error can't find the container with id 85c2798843abd3594b37119e9188654f94f1bf939b4a34862fa62d25d63d0b37 Oct 30 23:20:07.769385 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:07.769350 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" event={"ID":"145e7c01-efec-46e6-a748-56b39de76f83","Type":"ContainerStarted","Data":"0b7590c5d1b17285ac8a1ada673d7c46d3a6ec238a29c80a0518e93bb4e35798"} Oct 30 23:20:07.769539 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:07.769391 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" event={"ID":"145e7c01-efec-46e6-a748-56b39de76f83","Type":"ContainerStarted","Data":"85c2798843abd3594b37119e9188654f94f1bf939b4a34862fa62d25d63d0b37"} Oct 30 23:20:07.884982 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:07.884924 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" podStartSLOduration=1.884907318 podStartE2EDuration="1.884907318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:20:07.884590266 +0000 UTC m=+827.028420472" watchObservedRunningTime="2026-10-30 23:20:07.884907318 +0000 UTC m=+827.028737523" Oct 30 23:20:08.020707 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:08.020614 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:20:08.247785 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:08.247752 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" Oct 30 23:20:10.482344 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:10.482311 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:20:10.983495 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:10.983463 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb"] Oct 30 23:20:10.987158 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:10.987132 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" Oct 30 23:20:11.042003 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:11.041981 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b81a7a\"/\"openshift-service-ca.crt\"" Oct 30 23:20:11.058951 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:11.058920 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-tls-certs\") pod \"simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb\" (UID: \"e662ea1f-f4ae-43ff-852c-ed3d6d282c74\") " pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" Oct 30 23:20:11.059122 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:11.058965 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-home\") pod \"simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb\" (UID: \"e662ea1f-f4ae-43ff-852c-ed3d6d282c74\") " pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" Oct 30 23:20:11.059122 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:11.059085 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-dshm\") pod \"simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb\" (UID: \"e662ea1f-f4ae-43ff-852c-ed3d6d282c74\") " pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" Oct 30 23:20:11.059264 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:11.059126 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85cqg\" (UniqueName: \"kubernetes.io/projected/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-kube-api-access-85cqg\") pod \"simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb\" (UID: \"e662ea1f-f4ae-43ff-852c-ed3d6d282c74\") " pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" Oct 30 23:20:11.059264 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:11.059176 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-model-cache\") pod \"simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb\" (UID: \"e662ea1f-f4ae-43ff-852c-ed3d6d282c74\") " pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" Oct 30 23:20:11.059264 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:11.059243 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-tmp-dir\") pod \"simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb\" (UID: \"e662ea1f-f4ae-43ff-852c-ed3d6d282c74\") " pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" Oct 30 23:20:11.090723 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:11.090698 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b81a7a\"/\"kube-root-ca.crt\"" Oct 30 23:20:11.090841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:11.090725 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b81a7a\"/\"default-dockercfg-hwvw7\"" Oct 30 23:20:11.091205 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:11.091187 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b81a7a\"/\"simulator-w1-b81a7a-kserve-self-signed-certs\"" Oct 30 23:20:11.095218 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:11.095197 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb"] Oct 30 23:20:11.135935 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:11.135906 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:20:11.160081 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:11.160060 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-tls-certs\") pod \"simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb\" (UID: \"e662ea1f-f4ae-43ff-852c-ed3d6d282c74\") " pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" Oct 30 23:20:11.160199 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:11.160089 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-home\") pod \"simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb\" (UID: \"e662ea1f-f4ae-43ff-852c-ed3d6d282c74\") " pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" Oct 30 23:20:11.160199 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:11.160115 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-dshm\") pod \"simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb\" (UID: \"e662ea1f-f4ae-43ff-852c-ed3d6d282c74\") " pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" Oct 30 23:20:11.160199 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:11.160139 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85cqg\" (UniqueName: \"kubernetes.io/projected/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-kube-api-access-85cqg\") pod \"simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb\" (UID: \"e662ea1f-f4ae-43ff-852c-ed3d6d282c74\") " pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" Oct 30 23:20:11.160199 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:11.160159 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-model-cache\") pod \"simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb\" (UID: \"e662ea1f-f4ae-43ff-852c-ed3d6d282c74\") " pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" Oct 30 23:20:11.160419 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:11.160207 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-tmp-dir\") pod \"simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb\" (UID: \"e662ea1f-f4ae-43ff-852c-ed3d6d282c74\") " pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" Oct 30 23:20:11.160549 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:11.160530 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-home\") pod \"simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb\" (UID: \"e662ea1f-f4ae-43ff-852c-ed3d6d282c74\") " pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" Oct 30 23:20:11.160619 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:11.160598 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-tmp-dir\") pod \"simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb\" (UID: \"e662ea1f-f4ae-43ff-852c-ed3d6d282c74\") " pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" Oct 30 23:20:11.160657 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:11.160622 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-model-cache\") pod \"simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb\" (UID: \"e662ea1f-f4ae-43ff-852c-ed3d6d282c74\") " pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" Oct 30 23:20:11.162560 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:11.162537 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-dshm\") pod \"simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb\" (UID: \"e662ea1f-f4ae-43ff-852c-ed3d6d282c74\") " pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" Oct 30 23:20:11.162726 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:11.162710 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-tls-certs\") pod \"simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb\" (UID: \"e662ea1f-f4ae-43ff-852c-ed3d6d282c74\") " pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" Oct 30 23:20:11.217112 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:11.217093 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85cqg\" (UniqueName: \"kubernetes.io/projected/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-kube-api-access-85cqg\") pod \"simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb\" (UID: \"e662ea1f-f4ae-43ff-852c-ed3d6d282c74\") " pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" Oct 30 23:20:11.300706 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:11.300622 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" Oct 30 23:20:11.380026 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:11.379028 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:20:11.687692 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:11.687659 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb"] Oct 30 23:20:11.689354 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:20:11.689327 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode662ea1f_f4ae_43ff_852c_ed3d6d282c74.slice/crio-a9ff920794254c4706e9b411a6f8b5144517f325bec83a9d2dd1e4be75339bc4 WatchSource:0}: Error finding container a9ff920794254c4706e9b411a6f8b5144517f325bec83a9d2dd1e4be75339bc4: Status 404 returned error can't find the container with id a9ff920794254c4706e9b411a6f8b5144517f325bec83a9d2dd1e4be75339bc4 Oct 30 23:20:11.794154 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:11.794119 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" event={"ID":"e662ea1f-f4ae-43ff-852c-ed3d6d282c74","Type":"ContainerStarted","Data":"95c3dfecf98b9676e101eb7ffbebb99285ccb2779309883a2f70c501b0052016"} Oct 30 23:20:11.794317 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:11.794169 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" event={"ID":"e662ea1f-f4ae-43ff-852c-ed3d6d282c74","Type":"ContainerStarted","Data":"a9ff920794254c4706e9b411a6f8b5144517f325bec83a9d2dd1e4be75339bc4"} Oct 30 23:20:11.910395 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:11.910344 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" podStartSLOduration=1.910329878 podStartE2EDuration="1.910329878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:20:11.907681191 +0000 UTC m=+831.051511394" watchObservedRunningTime="2026-10-30 23:20:11.910329878 +0000 UTC m=+831.054160119" Oct 30 23:20:12.301520 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:12.301492 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" Oct 30 23:20:13.280895 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:13.280861 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:20:14.496779 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:14.496736 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:20:15.070478 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:15.070447 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:20:15.298660 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:15.298622 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:20:15.335188 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:15.335099 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:20:15.593095 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:15.593016 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:20:17.247194 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:17.247080 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" Oct 30 23:20:17.263836 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:17.260031 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" Oct 30 23:20:17.844330 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:17.844303 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" Oct 30 23:20:18.477105 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:18.477023 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:20:21.141876 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:20:21.141842 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:20:21.296858 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:21.296822 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:20:21.300764 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:21.300739 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" Oct 30 23:20:21.315484 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:21.315464 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" Oct 30 23:20:21.868128 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:21.868102 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" Oct 30 23:20:23.270468 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:23.270412 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:20:28.063593 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:28.063560 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:20:31.754799 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:31.754718 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll"] Oct 30 23:20:31.758972 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:31.758950 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" Oct 30 23:20:31.855769 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:31.855736 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43e39353-ae85-4d76-9900-7e7927409098-home\") pod \"simulator-w4-989a4f-kserve-6f68577976-kkpll\" (UID: \"43e39353-ae85-4d76-9900-7e7927409098\") " pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" Oct 30 23:20:31.855906 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:31.855814 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43e39353-ae85-4d76-9900-7e7927409098-tls-certs\") pod \"simulator-w4-989a4f-kserve-6f68577976-kkpll\" (UID: \"43e39353-ae85-4d76-9900-7e7927409098\") " pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" Oct 30 23:20:31.855906 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:31.855863 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43e39353-ae85-4d76-9900-7e7927409098-tmp-dir\") pod \"simulator-w4-989a4f-kserve-6f68577976-kkpll\" (UID: \"43e39353-ae85-4d76-9900-7e7927409098\") " pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" Oct 30 23:20:31.856027 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:31.855902 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43e39353-ae85-4d76-9900-7e7927409098-dshm\") pod \"simulator-w4-989a4f-kserve-6f68577976-kkpll\" (UID: \"43e39353-ae85-4d76-9900-7e7927409098\") " pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" Oct 30 23:20:31.856027 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:31.856017 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43e39353-ae85-4d76-9900-7e7927409098-model-cache\") pod \"simulator-w4-989a4f-kserve-6f68577976-kkpll\" (UID: \"43e39353-ae85-4d76-9900-7e7927409098\") " pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" Oct 30 23:20:31.856180 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:31.856088 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5wxm\" (UniqueName: \"kubernetes.io/projected/43e39353-ae85-4d76-9900-7e7927409098-kube-api-access-s5wxm\") pod \"simulator-w4-989a4f-kserve-6f68577976-kkpll\" (UID: \"43e39353-ae85-4d76-9900-7e7927409098\") " pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" Oct 30 23:20:31.861987 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:31.861964 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-989a4f\"/\"kube-root-ca.crt\"" Oct 30 23:20:31.861987 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:31.861983 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-989a4f\"/\"default-dockercfg-sr59g\"" Oct 30 23:20:31.862136 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:31.861983 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-989a4f\"/\"openshift-service-ca.crt\"" Oct 30 23:20:31.862416 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:31.862400 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-989a4f\"/\"simulator-w4-989a4f-kserve-self-signed-certs\"" Oct 30 23:20:31.864534 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:31.864517 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll"] Oct 30 23:20:31.933504 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:31.933469 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:20:31.957444 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:31.957418 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43e39353-ae85-4d76-9900-7e7927409098-tls-certs\") pod \"simulator-w4-989a4f-kserve-6f68577976-kkpll\" (UID: \"43e39353-ae85-4d76-9900-7e7927409098\") " pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" Oct 30 23:20:31.957567 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:31.957481 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43e39353-ae85-4d76-9900-7e7927409098-tmp-dir\") pod \"simulator-w4-989a4f-kserve-6f68577976-kkpll\" (UID: \"43e39353-ae85-4d76-9900-7e7927409098\") " pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" Oct 30 23:20:31.957567 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:31.957523 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43e39353-ae85-4d76-9900-7e7927409098-dshm\") pod \"simulator-w4-989a4f-kserve-6f68577976-kkpll\" (UID: \"43e39353-ae85-4d76-9900-7e7927409098\") " pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" Oct 30 23:20:31.957697 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:31.957598 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43e39353-ae85-4d76-9900-7e7927409098-model-cache\") pod \"simulator-w4-989a4f-kserve-6f68577976-kkpll\" (UID: \"43e39353-ae85-4d76-9900-7e7927409098\") " pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" Oct 30 23:20:31.957697 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:31.957635 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5wxm\" (UniqueName: \"kubernetes.io/projected/43e39353-ae85-4d76-9900-7e7927409098-kube-api-access-s5wxm\") pod \"simulator-w4-989a4f-kserve-6f68577976-kkpll\" (UID: \"43e39353-ae85-4d76-9900-7e7927409098\") " pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" Oct 30 23:20:31.957697 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:31.957683 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43e39353-ae85-4d76-9900-7e7927409098-home\") pod \"simulator-w4-989a4f-kserve-6f68577976-kkpll\" (UID: \"43e39353-ae85-4d76-9900-7e7927409098\") " pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" Oct 30 23:20:31.958054 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:31.958006 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43e39353-ae85-4d76-9900-7e7927409098-home\") pod \"simulator-w4-989a4f-kserve-6f68577976-kkpll\" (UID: \"43e39353-ae85-4d76-9900-7e7927409098\") " pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" Oct 30 23:20:31.958184 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:31.958161 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43e39353-ae85-4d76-9900-7e7927409098-tmp-dir\") pod \"simulator-w4-989a4f-kserve-6f68577976-kkpll\" (UID: \"43e39353-ae85-4d76-9900-7e7927409098\") " pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" Oct 30 23:20:31.958266 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:31.958258 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43e39353-ae85-4d76-9900-7e7927409098-model-cache\") pod \"simulator-w4-989a4f-kserve-6f68577976-kkpll\" (UID: \"43e39353-ae85-4d76-9900-7e7927409098\") " pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" Oct 30 23:20:31.960635 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:31.960611 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43e39353-ae85-4d76-9900-7e7927409098-dshm\") pod \"simulator-w4-989a4f-kserve-6f68577976-kkpll\" (UID: \"43e39353-ae85-4d76-9900-7e7927409098\") " pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" Oct 30 23:20:31.960908 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:31.960889 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43e39353-ae85-4d76-9900-7e7927409098-tls-certs\") pod \"simulator-w4-989a4f-kserve-6f68577976-kkpll\" (UID: \"43e39353-ae85-4d76-9900-7e7927409098\") " pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" Oct 30 23:20:32.015774 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:32.015710 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5wxm\" (UniqueName: \"kubernetes.io/projected/43e39353-ae85-4d76-9900-7e7927409098-kube-api-access-s5wxm\") pod \"simulator-w4-989a4f-kserve-6f68577976-kkpll\" (UID: \"43e39353-ae85-4d76-9900-7e7927409098\") " pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" Oct 30 23:20:32.097870 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:32.097840 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" Oct 30 23:20:32.477337 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:32.477305 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll"] Oct 30 23:20:32.477806 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:20:32.477784 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43e39353_ae85_4d76_9900_7e7927409098.slice/crio-dbab0483aeef4c494d22f349bd64d1f61766db3c24bf2dbd64c16e03748c8155 WatchSource:0}: Error finding container dbab0483aeef4c494d22f349bd64d1f61766db3c24bf2dbd64c16e03748c8155: Status 404 returned error can't find the container with id dbab0483aeef4c494d22f349bd64d1f61766db3c24bf2dbd64c16e03748c8155 Oct 30 23:20:32.905834 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:32.905797 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:20:32.924251 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:32.924220 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" event={"ID":"43e39353-ae85-4d76-9900-7e7927409098","Type":"ContainerStarted","Data":"70f31c88b1be0dd73610bd06a5d006b036cafec90cad1585d4560c958db9493d"} Oct 30 23:20:32.924405 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:32.924262 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" event={"ID":"43e39353-ae85-4d76-9900-7e7927409098","Type":"ContainerStarted","Data":"dbab0483aeef4c494d22f349bd64d1f61766db3c24bf2dbd64c16e03748c8155"} Oct 30 23:20:33.039371 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:33.039329 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" podStartSLOduration=2.039314014 podStartE2EDuration="2.039314014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:20:33.038442501 +0000 UTC m=+852.182272718" watchObservedRunningTime="2026-10-30 23:20:33.039314014 +0000 UTC m=+852.183144220" Oct 30 23:20:33.098358 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:33.098330 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" Oct 30 23:20:33.292081 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:33.291974 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:20:37.767221 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:37.767160 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:20:38.683199 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:38.683166 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:20:42.098760 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:42.098724 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" Oct 30 23:20:42.112623 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:42.112583 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" Oct 30 23:20:42.671582 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:42.671546 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:20:42.996811 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:42.996728 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" Oct 30 23:20:44.716088 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:44.716056 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:20:47.470915 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:47.470883 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:20:51.066867 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:51.066830 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:20:51.143088 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:20:51.143030 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:20:52.288439 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:52.288406 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:20:52.921413 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:52.921383 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d"] Oct 30 23:20:52.926368 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:52.926346 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" Oct 30 23:20:52.964798 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:52.964754 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-model-cache\") pod \"simulator-w3-2cfb80-kserve-7cf744bb95-frh9d\" (UID: \"da4935dd-c148-4cc4-84ab-1bfd91cce9b4\") " pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" Oct 30 23:20:52.964911 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:52.964802 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-home\") pod \"simulator-w3-2cfb80-kserve-7cf744bb95-frh9d\" (UID: \"da4935dd-c148-4cc4-84ab-1bfd91cce9b4\") " pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" Oct 30 23:20:52.964964 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:52.964917 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgdl2\" (UniqueName: \"kubernetes.io/projected/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-kube-api-access-fgdl2\") pod \"simulator-w3-2cfb80-kserve-7cf744bb95-frh9d\" (UID: \"da4935dd-c148-4cc4-84ab-1bfd91cce9b4\") " pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" Oct 30 23:20:52.964964 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:52.964951 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-dshm\") pod \"simulator-w3-2cfb80-kserve-7cf744bb95-frh9d\" (UID: \"da4935dd-c148-4cc4-84ab-1bfd91cce9b4\") " pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" Oct 30 23:20:52.965066 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:52.964969 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-tmp-dir\") pod \"simulator-w3-2cfb80-kserve-7cf744bb95-frh9d\" (UID: \"da4935dd-c148-4cc4-84ab-1bfd91cce9b4\") " pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" Oct 30 23:20:52.965066 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:52.964996 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-tls-certs\") pod \"simulator-w3-2cfb80-kserve-7cf744bb95-frh9d\" (UID: \"da4935dd-c148-4cc4-84ab-1bfd91cce9b4\") " pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" Oct 30 23:20:53.029205 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:53.029174 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2cfb80\"/\"kube-root-ca.crt\"" Oct 30 23:20:53.029336 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:53.029258 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2cfb80\"/\"default-dockercfg-9lpks\"" Oct 30 23:20:53.029336 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:53.029272 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2cfb80\"/\"openshift-service-ca.crt\"" Oct 30 23:20:53.029336 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:53.029290 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2cfb80\"/\"simulator-w3-2cfb80-kserve-self-signed-certs\"" Oct 30 23:20:53.032058 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:53.032025 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d"] Oct 30 23:20:53.065617 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:53.065590 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-model-cache\") pod \"simulator-w3-2cfb80-kserve-7cf744bb95-frh9d\" (UID: \"da4935dd-c148-4cc4-84ab-1bfd91cce9b4\") " pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" Oct 30 23:20:53.065772 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:53.065630 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-home\") pod \"simulator-w3-2cfb80-kserve-7cf744bb95-frh9d\" (UID: \"da4935dd-c148-4cc4-84ab-1bfd91cce9b4\") " pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" Oct 30 23:20:53.065772 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:53.065732 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgdl2\" (UniqueName: \"kubernetes.io/projected/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-kube-api-access-fgdl2\") pod \"simulator-w3-2cfb80-kserve-7cf744bb95-frh9d\" (UID: \"da4935dd-c148-4cc4-84ab-1bfd91cce9b4\") " pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" Oct 30 23:20:53.065893 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:53.065770 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-dshm\") pod \"simulator-w3-2cfb80-kserve-7cf744bb95-frh9d\" (UID: \"da4935dd-c148-4cc4-84ab-1bfd91cce9b4\") " pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" Oct 30 23:20:53.065893 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:53.065794 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-tmp-dir\") pod \"simulator-w3-2cfb80-kserve-7cf744bb95-frh9d\" (UID: \"da4935dd-c148-4cc4-84ab-1bfd91cce9b4\") " pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" Oct 30 23:20:53.065893 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:53.065837 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-tls-certs\") pod \"simulator-w3-2cfb80-kserve-7cf744bb95-frh9d\" (UID: \"da4935dd-c148-4cc4-84ab-1bfd91cce9b4\") " pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" Oct 30 23:20:53.066980 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:53.066941 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-model-cache\") pod \"simulator-w3-2cfb80-kserve-7cf744bb95-frh9d\" (UID: \"da4935dd-c148-4cc4-84ab-1bfd91cce9b4\") " pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" Oct 30 23:20:53.067320 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:53.067237 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-tmp-dir\") pod \"simulator-w3-2cfb80-kserve-7cf744bb95-frh9d\" (UID: \"da4935dd-c148-4cc4-84ab-1bfd91cce9b4\") " pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" Oct 30 23:20:53.067320 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:53.067256 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-home\") pod \"simulator-w3-2cfb80-kserve-7cf744bb95-frh9d\" (UID: \"da4935dd-c148-4cc4-84ab-1bfd91cce9b4\") " pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" Oct 30 23:20:53.069080 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:53.069058 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-tls-certs\") pod \"simulator-w3-2cfb80-kserve-7cf744bb95-frh9d\" (UID: \"da4935dd-c148-4cc4-84ab-1bfd91cce9b4\") " pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" Oct 30 23:20:53.069912 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:53.069888 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-dshm\") pod \"simulator-w3-2cfb80-kserve-7cf744bb95-frh9d\" (UID: \"da4935dd-c148-4cc4-84ab-1bfd91cce9b4\") " pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" Oct 30 23:20:53.094930 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:53.094901 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:20:53.124624 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:53.124597 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgdl2\" (UniqueName: \"kubernetes.io/projected/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-kube-api-access-fgdl2\") pod \"simulator-w3-2cfb80-kserve-7cf744bb95-frh9d\" (UID: \"da4935dd-c148-4cc4-84ab-1bfd91cce9b4\") " pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" Oct 30 23:20:53.239091 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:53.238994 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" Oct 30 23:20:53.618664 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:53.618639 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d"] Oct 30 23:20:53.619634 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:20:53.619607 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda4935dd_c148_4cc4_84ab_1bfd91cce9b4.slice/crio-5db4c0780e9e152107889799c3363a455d4c0d06e53b4060fe7e291b1a6c834a WatchSource:0}: Error finding container 5db4c0780e9e152107889799c3363a455d4c0d06e53b4060fe7e291b1a6c834a: Status 404 returned error can't find the container with id 5db4c0780e9e152107889799c3363a455d4c0d06e53b4060fe7e291b1a6c834a Oct 30 23:20:54.053472 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:54.053388 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" event={"ID":"da4935dd-c148-4cc4-84ab-1bfd91cce9b4","Type":"ContainerStarted","Data":"b8523a64b94bc7aec9fed2c260c618b394a0c4217df92c65adc3da67b70f94a9"} Oct 30 23:20:54.053472 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:54.053427 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" event={"ID":"da4935dd-c148-4cc4-84ab-1bfd91cce9b4","Type":"ContainerStarted","Data":"5db4c0780e9e152107889799c3363a455d4c0d06e53b4060fe7e291b1a6c834a"} Oct 30 23:20:54.180621 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:54.180569 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" podStartSLOduration=2.180556523 podStartE2EDuration="2.180556523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:20:54.178400274 +0000 UTC m=+873.322230481" watchObservedRunningTime="2026-10-30 23:20:54.180556523 +0000 UTC m=+873.324386728" Oct 30 23:20:54.239753 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:54.239733 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" Oct 30 23:20:57.364194 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:57.364159 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:20:57.608989 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:20:57.608941 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:21:02.234993 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:02.234948 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:21:03.239552 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:03.239514 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" Oct 30 23:21:03.256091 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:03.256051 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" Oct 30 23:21:04.126773 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:04.126738 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" Oct 30 23:21:07.561818 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:07.561750 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:21:12.381157 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:12.381120 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:21:13.105550 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:13.105525 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:21:15.510219 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:15.510184 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:21:17.555507 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:17.555471 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf"] Oct 30 23:21:17.559331 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:17.559316 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" Oct 30 23:21:17.576533 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:17.576505 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:21:17.622382 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:17.622361 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-aadaa6\"/\"premium-w6-aadaa6-kserve-self-signed-certs\"" Oct 30 23:21:17.674850 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:17.674823 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf"] Oct 30 23:21:17.697065 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:17.697006 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/959b92af-9035-4dbd-aed3-5db4daaa63a8-dshm\") pod \"premium-w6-aadaa6-kserve-8586b84766-zddpf\" (UID: \"959b92af-9035-4dbd-aed3-5db4daaa63a8\") " pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" Oct 30 23:21:17.697228 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:17.697102 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/959b92af-9035-4dbd-aed3-5db4daaa63a8-tls-certs\") pod \"premium-w6-aadaa6-kserve-8586b84766-zddpf\" (UID: \"959b92af-9035-4dbd-aed3-5db4daaa63a8\") " pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" Oct 30 23:21:17.697228 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:17.697142 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsqpr\" (UniqueName: \"kubernetes.io/projected/959b92af-9035-4dbd-aed3-5db4daaa63a8-kube-api-access-tsqpr\") pod \"premium-w6-aadaa6-kserve-8586b84766-zddpf\" (UID: \"959b92af-9035-4dbd-aed3-5db4daaa63a8\") " pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" Oct 30 23:21:17.697364 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:17.697242 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/959b92af-9035-4dbd-aed3-5db4daaa63a8-tmp-dir\") pod \"premium-w6-aadaa6-kserve-8586b84766-zddpf\" (UID: \"959b92af-9035-4dbd-aed3-5db4daaa63a8\") " pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" Oct 30 23:21:17.697364 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:17.697353 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/959b92af-9035-4dbd-aed3-5db4daaa63a8-model-cache\") pod \"premium-w6-aadaa6-kserve-8586b84766-zddpf\" (UID: \"959b92af-9035-4dbd-aed3-5db4daaa63a8\") " pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" Oct 30 23:21:17.697473 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:17.697387 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/959b92af-9035-4dbd-aed3-5db4daaa63a8-home\") pod \"premium-w6-aadaa6-kserve-8586b84766-zddpf\" (UID: \"959b92af-9035-4dbd-aed3-5db4daaa63a8\") " pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" Oct 30 23:21:17.798391 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:17.798359 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/959b92af-9035-4dbd-aed3-5db4daaa63a8-model-cache\") pod \"premium-w6-aadaa6-kserve-8586b84766-zddpf\" (UID: \"959b92af-9035-4dbd-aed3-5db4daaa63a8\") " pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" Oct 30 23:21:17.798553 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:17.798406 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/959b92af-9035-4dbd-aed3-5db4daaa63a8-home\") pod \"premium-w6-aadaa6-kserve-8586b84766-zddpf\" (UID: \"959b92af-9035-4dbd-aed3-5db4daaa63a8\") " pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" Oct 30 23:21:17.798553 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:17.798525 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/959b92af-9035-4dbd-aed3-5db4daaa63a8-dshm\") pod \"premium-w6-aadaa6-kserve-8586b84766-zddpf\" (UID: \"959b92af-9035-4dbd-aed3-5db4daaa63a8\") " pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" Oct 30 23:21:17.798678 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:17.798596 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/959b92af-9035-4dbd-aed3-5db4daaa63a8-tls-certs\") pod \"premium-w6-aadaa6-kserve-8586b84766-zddpf\" (UID: \"959b92af-9035-4dbd-aed3-5db4daaa63a8\") " pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" Oct 30 23:21:17.798678 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:17.798617 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsqpr\" (UniqueName: \"kubernetes.io/projected/959b92af-9035-4dbd-aed3-5db4daaa63a8-kube-api-access-tsqpr\") pod \"premium-w6-aadaa6-kserve-8586b84766-zddpf\" (UID: \"959b92af-9035-4dbd-aed3-5db4daaa63a8\") " pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" Oct 30 23:21:17.798678 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:17.798639 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/959b92af-9035-4dbd-aed3-5db4daaa63a8-tmp-dir\") pod \"premium-w6-aadaa6-kserve-8586b84766-zddpf\" (UID: \"959b92af-9035-4dbd-aed3-5db4daaa63a8\") " pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" Oct 30 23:21:17.798901 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:17.798856 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/959b92af-9035-4dbd-aed3-5db4daaa63a8-model-cache\") pod \"premium-w6-aadaa6-kserve-8586b84766-zddpf\" (UID: \"959b92af-9035-4dbd-aed3-5db4daaa63a8\") " pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" Oct 30 23:21:17.798990 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:17.798896 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/959b92af-9035-4dbd-aed3-5db4daaa63a8-home\") pod \"premium-w6-aadaa6-kserve-8586b84766-zddpf\" (UID: \"959b92af-9035-4dbd-aed3-5db4daaa63a8\") " pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" Oct 30 23:21:17.799081 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:17.799063 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/959b92af-9035-4dbd-aed3-5db4daaa63a8-tmp-dir\") pod \"premium-w6-aadaa6-kserve-8586b84766-zddpf\" (UID: \"959b92af-9035-4dbd-aed3-5db4daaa63a8\") " pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" Oct 30 23:21:17.801291 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:17.801270 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/959b92af-9035-4dbd-aed3-5db4daaa63a8-dshm\") pod \"premium-w6-aadaa6-kserve-8586b84766-zddpf\" (UID: \"959b92af-9035-4dbd-aed3-5db4daaa63a8\") " pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" Oct 30 23:21:17.801376 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:17.801269 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/959b92af-9035-4dbd-aed3-5db4daaa63a8-tls-certs\") pod \"premium-w6-aadaa6-kserve-8586b84766-zddpf\" (UID: \"959b92af-9035-4dbd-aed3-5db4daaa63a8\") " pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" Oct 30 23:21:17.859134 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:17.859108 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsqpr\" (UniqueName: \"kubernetes.io/projected/959b92af-9035-4dbd-aed3-5db4daaa63a8-kube-api-access-tsqpr\") pod \"premium-w6-aadaa6-kserve-8586b84766-zddpf\" (UID: \"959b92af-9035-4dbd-aed3-5db4daaa63a8\") " pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" Oct 30 23:21:17.871770 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:17.871743 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" Oct 30 23:21:18.278224 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:18.278198 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf"] Oct 30 23:21:18.278678 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:21:18.278658 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod959b92af_9035_4dbd_aed3_5db4daaa63a8.slice/crio-6e4f9222dfdde8e0e77223460ad74bebbdb111282c597847db3a7d052e05d0ed WatchSource:0}: Error finding container 6e4f9222dfdde8e0e77223460ad74bebbdb111282c597847db3a7d052e05d0ed: Status 404 returned error can't find the container with id 6e4f9222dfdde8e0e77223460ad74bebbdb111282c597847db3a7d052e05d0ed Oct 30 23:21:19.017396 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:19.017358 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:21:19.210121 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:19.210081 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" event={"ID":"959b92af-9035-4dbd-aed3-5db4daaa63a8","Type":"ContainerStarted","Data":"9f7156db0d849717653d4f2107439baedc5f52f49bd3109c08037709775c77c4"} Oct 30 23:21:19.210302 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:19.210139 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" event={"ID":"959b92af-9035-4dbd-aed3-5db4daaa63a8","Type":"ContainerStarted","Data":"6e4f9222dfdde8e0e77223460ad74bebbdb111282c597847db3a7d052e05d0ed"} Oct 30 23:21:19.328571 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:19.328484 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" podStartSLOduration=2.328470969 podStartE2EDuration="2.328470969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:21:19.325370708 +0000 UTC m=+898.469200914" watchObservedRunningTime="2026-10-30 23:21:19.328470969 +0000 UTC m=+898.472301174" Oct 30 23:21:19.872886 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:19.872860 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" Oct 30 23:21:21.066096 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:21.066067 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft_91651ce2-4673-4ced-b26d-e94ff8829d11/extract/0.log" Oct 30 23:21:21.068548 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:21.068529 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft_91651ce2-4673-4ced-b26d-e94ff8829d11/extract/0.log" Oct 30 23:21:21.076470 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:21.076451 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/2.log" Oct 30 23:21:21.079047 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:21.079016 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/2.log" Oct 30 23:21:21.082507 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:21.082490 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 30 23:21:21.085079 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:21.085063 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 30 23:21:21.142349 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:21:21.142319 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:21:22.650319 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:22.650282 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:21:27.425508 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:27.425470 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:21:27.872445 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:27.872417 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" Oct 30 23:21:27.885631 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:27.885605 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" Oct 30 23:21:28.275844 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:28.275776 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" Oct 30 23:21:32.339202 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:32.339171 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:21:37.201818 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:37.201776 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:21:41.734683 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:41.734649 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:21:42.097090 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:42.097062 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:21:44.762904 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:44.762871 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:21:46.978879 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:46.978841 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:21:51.143802 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:21:51.143759 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:21:51.804673 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:51.804625 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:21:52.045507 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:52.045463 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:21:56.675442 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:56.675400 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:21:57.093659 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:57.093630 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:21:57.426173 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:21:57.426092 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:22:01.584657 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:01.584571 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:22:03.691930 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:03.691897 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg"] Oct 30 23:22:03.695766 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:03.695750 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" Oct 30 23:22:03.798925 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:03.798898 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-989a4f\"/\"premium-w4-989a4f-kserve-self-signed-certs\"" Oct 30 23:22:03.801689 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:03.801668 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg"] Oct 30 23:22:03.838169 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:03.838127 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44adfd6a-2626-4378-bca1-d46ee1670b07-tmp-dir\") pod \"premium-w4-989a4f-kserve-5bdb85b789-s4xcg\" (UID: \"44adfd6a-2626-4378-bca1-d46ee1670b07\") " pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" Oct 30 23:22:03.838351 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:03.838188 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44adfd6a-2626-4378-bca1-d46ee1670b07-home\") pod \"premium-w4-989a4f-kserve-5bdb85b789-s4xcg\" (UID: \"44adfd6a-2626-4378-bca1-d46ee1670b07\") " pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" Oct 30 23:22:03.838351 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:03.838238 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlfks\" (UniqueName: \"kubernetes.io/projected/44adfd6a-2626-4378-bca1-d46ee1670b07-kube-api-access-dlfks\") pod \"premium-w4-989a4f-kserve-5bdb85b789-s4xcg\" (UID: \"44adfd6a-2626-4378-bca1-d46ee1670b07\") " pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" Oct 30 23:22:03.838351 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:03.838269 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44adfd6a-2626-4378-bca1-d46ee1670b07-dshm\") pod \"premium-w4-989a4f-kserve-5bdb85b789-s4xcg\" (UID: \"44adfd6a-2626-4378-bca1-d46ee1670b07\") " pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" Oct 30 23:22:03.838525 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:03.838356 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44adfd6a-2626-4378-bca1-d46ee1670b07-tls-certs\") pod \"premium-w4-989a4f-kserve-5bdb85b789-s4xcg\" (UID: \"44adfd6a-2626-4378-bca1-d46ee1670b07\") " pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" Oct 30 23:22:03.838525 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:03.838425 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44adfd6a-2626-4378-bca1-d46ee1670b07-model-cache\") pod \"premium-w4-989a4f-kserve-5bdb85b789-s4xcg\" (UID: \"44adfd6a-2626-4378-bca1-d46ee1670b07\") " pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" Oct 30 23:22:03.856342 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:03.856307 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:22:03.939819 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:03.939787 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44adfd6a-2626-4378-bca1-d46ee1670b07-home\") pod \"premium-w4-989a4f-kserve-5bdb85b789-s4xcg\" (UID: \"44adfd6a-2626-4378-bca1-d46ee1670b07\") " pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" Oct 30 23:22:03.939995 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:03.939834 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlfks\" (UniqueName: \"kubernetes.io/projected/44adfd6a-2626-4378-bca1-d46ee1670b07-kube-api-access-dlfks\") pod \"premium-w4-989a4f-kserve-5bdb85b789-s4xcg\" (UID: \"44adfd6a-2626-4378-bca1-d46ee1670b07\") " pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" Oct 30 23:22:03.939995 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:03.939860 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44adfd6a-2626-4378-bca1-d46ee1670b07-dshm\") pod \"premium-w4-989a4f-kserve-5bdb85b789-s4xcg\" (UID: \"44adfd6a-2626-4378-bca1-d46ee1670b07\") " pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" Oct 30 23:22:03.939995 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:03.939936 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44adfd6a-2626-4378-bca1-d46ee1670b07-tls-certs\") pod \"premium-w4-989a4f-kserve-5bdb85b789-s4xcg\" (UID: \"44adfd6a-2626-4378-bca1-d46ee1670b07\") " pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" Oct 30 23:22:03.940230 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:03.940074 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44adfd6a-2626-4378-bca1-d46ee1670b07-model-cache\") pod \"premium-w4-989a4f-kserve-5bdb85b789-s4xcg\" (UID: \"44adfd6a-2626-4378-bca1-d46ee1670b07\") " pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" Oct 30 23:22:03.940230 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:03.940175 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44adfd6a-2626-4378-bca1-d46ee1670b07-tmp-dir\") pod \"premium-w4-989a4f-kserve-5bdb85b789-s4xcg\" (UID: \"44adfd6a-2626-4378-bca1-d46ee1670b07\") " pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" Oct 30 23:22:03.940351 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:03.940244 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44adfd6a-2626-4378-bca1-d46ee1670b07-home\") pod \"premium-w4-989a4f-kserve-5bdb85b789-s4xcg\" (UID: \"44adfd6a-2626-4378-bca1-d46ee1670b07\") " pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" Oct 30 23:22:03.940470 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:03.940444 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44adfd6a-2626-4378-bca1-d46ee1670b07-tmp-dir\") pod \"premium-w4-989a4f-kserve-5bdb85b789-s4xcg\" (UID: \"44adfd6a-2626-4378-bca1-d46ee1670b07\") " pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" Oct 30 23:22:03.940470 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:03.940448 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44adfd6a-2626-4378-bca1-d46ee1670b07-model-cache\") pod \"premium-w4-989a4f-kserve-5bdb85b789-s4xcg\" (UID: \"44adfd6a-2626-4378-bca1-d46ee1670b07\") " pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" Oct 30 23:22:03.942432 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:03.942376 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44adfd6a-2626-4378-bca1-d46ee1670b07-dshm\") pod \"premium-w4-989a4f-kserve-5bdb85b789-s4xcg\" (UID: \"44adfd6a-2626-4378-bca1-d46ee1670b07\") " pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" Oct 30 23:22:03.942664 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:03.942646 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44adfd6a-2626-4378-bca1-d46ee1670b07-tls-certs\") pod \"premium-w4-989a4f-kserve-5bdb85b789-s4xcg\" (UID: \"44adfd6a-2626-4378-bca1-d46ee1670b07\") " pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" Oct 30 23:22:03.997065 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:03.997002 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlfks\" (UniqueName: \"kubernetes.io/projected/44adfd6a-2626-4378-bca1-d46ee1670b07-kube-api-access-dlfks\") pod \"premium-w4-989a4f-kserve-5bdb85b789-s4xcg\" (UID: \"44adfd6a-2626-4378-bca1-d46ee1670b07\") " pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" Oct 30 23:22:04.008292 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:04.008270 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" Oct 30 23:22:04.404357 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:04.404326 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg"] Oct 30 23:22:04.406594 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:22:04.406556 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44adfd6a_2626_4378_bca1_d46ee1670b07.slice/crio-4419089e86a9934100686a503163f262d716d104cb84d92fe8db20f1b4883a9c WatchSource:0}: Error finding container 4419089e86a9934100686a503163f262d716d104cb84d92fe8db20f1b4883a9c: Status 404 returned error can't find the container with id 4419089e86a9934100686a503163f262d716d104cb84d92fe8db20f1b4883a9c Oct 30 23:22:04.494925 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:04.494891 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" event={"ID":"44adfd6a-2626-4378-bca1-d46ee1670b07","Type":"ContainerStarted","Data":"1654b012d1b730807798a9ab3f575b0c4504c87d2e0b5af0dac47148aaf91c17"} Oct 30 23:22:04.495093 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:04.494942 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" event={"ID":"44adfd6a-2626-4378-bca1-d46ee1670b07","Type":"ContainerStarted","Data":"4419089e86a9934100686a503163f262d716d104cb84d92fe8db20f1b4883a9c"} Oct 30 23:22:04.608858 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:04.608809 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" podStartSLOduration=1.6087938290000001 podStartE2EDuration="1.608793829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:22:04.607772895 +0000 UTC m=+943.751603121" watchObservedRunningTime="2026-10-30 23:22:04.608793829 +0000 UTC m=+943.752624033" Oct 30 23:22:04.972713 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:04.972685 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:22:05.009059 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:05.008675 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" Oct 30 23:22:06.386534 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:06.386492 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:22:11.171374 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:11.171340 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:22:14.009229 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:14.009183 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" Oct 30 23:22:14.021481 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:14.021457 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" Oct 30 23:22:14.570386 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:14.570360 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" Oct 30 23:22:15.829752 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:15.829715 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:22:16.079605 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:16.079569 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:22:17.080617 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:17.080547 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:22:20.992086 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:20.992023 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:22:21.142191 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:22:21.142158 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:22:25.787224 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:25.787169 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:22:26.916226 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:26.916190 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:22:30.697903 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:30.697833 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:22:35.388705 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:35.388668 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:22:36.686378 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:36.686343 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:22:38.076366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:38.076336 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh"] Oct 30 23:22:38.080299 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:38.080279 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" Oct 30 23:22:38.178270 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:38.178241 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-home\") pod \"premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh\" (UID: \"6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7\") " pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" Oct 30 23:22:38.178433 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:38.178278 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-tls-certs\") pod \"premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh\" (UID: \"6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7\") " pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" Oct 30 23:22:38.178433 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:38.178296 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-dshm\") pod \"premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh\" (UID: \"6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7\") " pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" Oct 30 23:22:38.178433 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:38.178383 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2m9c\" (UniqueName: \"kubernetes.io/projected/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-kube-api-access-t2m9c\") pod \"premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh\" (UID: \"6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7\") " pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" Oct 30 23:22:38.178598 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:38.178453 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-tmp-dir\") pod \"premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh\" (UID: \"6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7\") " pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" Oct 30 23:22:38.178598 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:38.178515 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-model-cache\") pod \"premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh\" (UID: \"6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7\") " pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" Oct 30 23:22:38.183515 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:38.183495 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-5dc3c0\"/\"openshift-service-ca.crt\"" Oct 30 23:22:38.183671 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:38.183655 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-5dc3c0\"/\"kube-root-ca.crt\"" Oct 30 23:22:38.183788 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:38.183776 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-5dc3c0\"/\"premium-w5-5dc3c0-kserve-self-signed-certs\"" Oct 30 23:22:38.183902 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:38.183889 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-5dc3c0\"/\"default-dockercfg-jnqrc\"" Oct 30 23:22:38.186336 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:38.186319 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh"] Oct 30 23:22:38.279240 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:38.279192 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-tls-certs\") pod \"premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh\" (UID: \"6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7\") " pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" Oct 30 23:22:38.279240 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:38.279238 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-dshm\") pod \"premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh\" (UID: \"6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7\") " pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" Oct 30 23:22:38.279551 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:38.279313 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2m9c\" (UniqueName: \"kubernetes.io/projected/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-kube-api-access-t2m9c\") pod \"premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh\" (UID: \"6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7\") " pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" Oct 30 23:22:38.279551 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:38.279383 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-tmp-dir\") pod \"premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh\" (UID: \"6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7\") " pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" Oct 30 23:22:38.279551 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:38.279410 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-model-cache\") pod \"premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh\" (UID: \"6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7\") " pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" Oct 30 23:22:38.279551 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:38.279503 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-home\") pod \"premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh\" (UID: \"6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7\") " pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" Oct 30 23:22:38.279844 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:38.279813 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-tmp-dir\") pod \"premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh\" (UID: \"6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7\") " pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" Oct 30 23:22:38.279963 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:38.279934 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:22:38.280233 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:38.280212 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-model-cache\") pod \"premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh\" (UID: \"6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7\") " pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" Oct 30 23:22:38.280619 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:38.280597 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-home\") pod \"premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh\" (UID: \"6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7\") " pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" Oct 30 23:22:38.282005 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:38.281980 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-dshm\") pod \"premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh\" (UID: \"6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7\") " pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" Oct 30 23:22:38.282200 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:38.282184 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-tls-certs\") pod \"premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh\" (UID: \"6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7\") " pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" Oct 30 23:22:38.337629 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:38.337604 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2m9c\" (UniqueName: \"kubernetes.io/projected/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-kube-api-access-t2m9c\") pod \"premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh\" (UID: \"6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7\") " pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" Oct 30 23:22:38.393090 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:38.393052 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" Oct 30 23:22:38.794499 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:38.794471 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh"] Oct 30 23:22:38.795725 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:22:38.795699 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c4bf3c4_70de_4d3a_bef9_d23aa84bcfe7.slice/crio-6fff41e39b21fd6723f5f17bec85564d892f04e84e8699c21f0c84a065243b1c WatchSource:0}: Error finding container 6fff41e39b21fd6723f5f17bec85564d892f04e84e8699c21f0c84a065243b1c: Status 404 returned error can't find the container with id 6fff41e39b21fd6723f5f17bec85564d892f04e84e8699c21f0c84a065243b1c Oct 30 23:22:39.712903 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:39.712871 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" event={"ID":"6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7","Type":"ContainerStarted","Data":"67fd5b2ee33c17b620637500c422df0e4e72201dae1572812746117d1ba279ec"} Oct 30 23:22:39.712903 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:39.712916 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" event={"ID":"6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7","Type":"ContainerStarted","Data":"6fff41e39b21fd6723f5f17bec85564d892f04e84e8699c21f0c84a065243b1c"} Oct 30 23:22:39.828451 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:39.828396 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" podStartSLOduration=1.828382226 podStartE2EDuration="1.828382226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:22:39.826751022 +0000 UTC m=+978.970581224" watchObservedRunningTime="2026-10-30 23:22:39.828382226 +0000 UTC m=+978.972212431" Oct 30 23:22:40.371323 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:40.371281 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:22:40.393324 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:40.393301 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" Oct 30 23:22:45.209701 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:45.209660 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:22:48.393548 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:48.393519 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" Oct 30 23:22:48.406130 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:48.406105 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" Oct 30 23:22:48.788900 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:48.788816 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" Oct 30 23:22:50.097322 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:50.097285 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:22:51.152517 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:22:51.152475 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:22:54.874408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:54.874344 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:22:59.684487 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:22:59.684437 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:23:04.083055 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:04.083008 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:23:04.508581 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:04.508546 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:23:09.377841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:09.377777 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:23:14.180306 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:14.180272 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:23:16.173717 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:16.173672 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:23:16.405643 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:16.405609 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:23:19.060598 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:19.060555 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:23:21.142974 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:23:21.142933 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:23:23.888995 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:23.888964 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:23:28.667959 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:28.667928 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:23:33.513300 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:33.513227 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:23:38.305220 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:38.305189 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:23:43.053239 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:43.053205 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:23:47.880442 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:47.880408 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:23:51.142564 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:23:51.142524 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:23:52.715361 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:52.715327 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:23:52.941732 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:52.941699 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:23:54.746764 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:54.746734 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb"] Oct 30 23:23:54.750819 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:54.750800 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" Oct 30 23:23:54.778141 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:54.778106 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee837874-e627-482a-8b34-19c8c5b55fc5-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6zvgb\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" Oct 30 23:23:54.778282 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:54.778191 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee837874-e627-482a-8b34-19c8c5b55fc5-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6zvgb\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" Oct 30 23:23:54.778282 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:54.778231 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mhn5\" (UniqueName: \"kubernetes.io/projected/ee837874-e627-482a-8b34-19c8c5b55fc5-kube-api-access-9mhn5\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6zvgb\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" Oct 30 23:23:54.778393 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:54.778325 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee837874-e627-482a-8b34-19c8c5b55fc5-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6zvgb\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" Oct 30 23:23:54.778393 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:54.778376 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee837874-e627-482a-8b34-19c8c5b55fc5-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6zvgb\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" Oct 30 23:23:54.778495 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:54.778420 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee837874-e627-482a-8b34-19c8c5b55fc5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6zvgb\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" Oct 30 23:23:54.778495 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:54.778458 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee837874-e627-482a-8b34-19c8c5b55fc5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6zvgb\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" Oct 30 23:23:54.778645 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:54.778619 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee837874-e627-482a-8b34-19c8c5b55fc5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6zvgb\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" Oct 30 23:23:54.778707 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:54.778665 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee837874-e627-482a-8b34-19c8c5b55fc5-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6zvgb\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" Oct 30 23:23:54.849226 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:54.848949 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb"] Oct 30 23:23:54.881978 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:54.880382 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee837874-e627-482a-8b34-19c8c5b55fc5-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6zvgb\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" Oct 30 23:23:54.881978 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:54.880467 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee837874-e627-482a-8b34-19c8c5b55fc5-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6zvgb\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" Oct 30 23:23:54.881978 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:54.880511 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mhn5\" (UniqueName: \"kubernetes.io/projected/ee837874-e627-482a-8b34-19c8c5b55fc5-kube-api-access-9mhn5\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6zvgb\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" Oct 30 23:23:54.881978 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:54.880546 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee837874-e627-482a-8b34-19c8c5b55fc5-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6zvgb\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" Oct 30 23:23:54.881978 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:54.880577 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee837874-e627-482a-8b34-19c8c5b55fc5-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6zvgb\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" Oct 30 23:23:54.881978 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:54.880616 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee837874-e627-482a-8b34-19c8c5b55fc5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6zvgb\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" Oct 30 23:23:54.881978 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:54.880651 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee837874-e627-482a-8b34-19c8c5b55fc5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6zvgb\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" Oct 30 23:23:54.881978 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:54.880708 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee837874-e627-482a-8b34-19c8c5b55fc5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6zvgb\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" Oct 30 23:23:54.881978 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:54.880748 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee837874-e627-482a-8b34-19c8c5b55fc5-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6zvgb\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" Oct 30 23:23:54.881978 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:54.881242 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee837874-e627-482a-8b34-19c8c5b55fc5-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6zvgb\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" Oct 30 23:23:54.881978 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:54.881506 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee837874-e627-482a-8b34-19c8c5b55fc5-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6zvgb\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" Oct 30 23:23:54.881978 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:54.881741 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee837874-e627-482a-8b34-19c8c5b55fc5-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6zvgb\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" Oct 30 23:23:54.882798 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:54.882399 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee837874-e627-482a-8b34-19c8c5b55fc5-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6zvgb\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" Oct 30 23:23:54.882858 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:54.882815 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee837874-e627-482a-8b34-19c8c5b55fc5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6zvgb\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" Oct 30 23:23:54.885159 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:54.885128 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee837874-e627-482a-8b34-19c8c5b55fc5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6zvgb\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" Oct 30 23:23:54.885611 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:54.885586 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee837874-e627-482a-8b34-19c8c5b55fc5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6zvgb\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" Oct 30 23:23:54.938853 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:54.938814 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee837874-e627-482a-8b34-19c8c5b55fc5-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6zvgb\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" Oct 30 23:23:54.938966 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:54.938887 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mhn5\" (UniqueName: \"kubernetes.io/projected/ee837874-e627-482a-8b34-19c8c5b55fc5-kube-api-access-9mhn5\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6zvgb\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" Oct 30 23:23:55.000723 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:55.000664 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:23:55.063986 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:55.063958 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" Oct 30 23:23:55.203829 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:55.203801 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cc6b31f-npwbj"] Oct 30 23:23:55.208714 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:55.208697 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc6b31f-npwbj" Oct 30 23:23:55.261305 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:55.261247 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:23:55.261464 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:55.261433 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\"" Oct 30 23:23:55.263990 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:55.263964 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cc6b31f-npwbj"] Oct 30 23:23:55.285470 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:55.285445 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/becfdd7d-d2f1-4a08-aeca-275a4a0e40b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cc6b31f-npwbj\" (UID: \"becfdd7d-d2f1-4a08-aeca-275a4a0e40b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc6b31f-npwbj" Oct 30 23:23:55.285609 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:55.285509 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/becfdd7d-d2f1-4a08-aeca-275a4a0e40b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cc6b31f-npwbj\" (UID: \"becfdd7d-d2f1-4a08-aeca-275a4a0e40b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc6b31f-npwbj" Oct 30 23:23:55.285676 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:55.285625 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74r4m\" (UniqueName: \"kubernetes.io/projected/becfdd7d-d2f1-4a08-aeca-275a4a0e40b2-kube-api-access-74r4m\") pod \"maas-api-revoke-sub-0cc6b31f-npwbj\" (UID: \"becfdd7d-d2f1-4a08-aeca-275a4a0e40b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc6b31f-npwbj" Oct 30 23:23:55.386769 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:55.386744 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/becfdd7d-d2f1-4a08-aeca-275a4a0e40b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cc6b31f-npwbj\" (UID: \"becfdd7d-d2f1-4a08-aeca-275a4a0e40b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc6b31f-npwbj" Oct 30 23:23:55.386900 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:55.386793 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/becfdd7d-d2f1-4a08-aeca-275a4a0e40b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cc6b31f-npwbj\" (UID: \"becfdd7d-d2f1-4a08-aeca-275a4a0e40b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc6b31f-npwbj" Oct 30 23:23:55.386900 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:55.386821 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74r4m\" (UniqueName: \"kubernetes.io/projected/becfdd7d-d2f1-4a08-aeca-275a4a0e40b2-kube-api-access-74r4m\") pod \"maas-api-revoke-sub-0cc6b31f-npwbj\" (UID: \"becfdd7d-d2f1-4a08-aeca-275a4a0e40b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc6b31f-npwbj" Oct 30 23:23:55.387551 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:55.387502 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/becfdd7d-d2f1-4a08-aeca-275a4a0e40b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cc6b31f-npwbj\" (UID: \"becfdd7d-d2f1-4a08-aeca-275a4a0e40b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc6b31f-npwbj" Oct 30 23:23:55.389435 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:55.389412 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/becfdd7d-d2f1-4a08-aeca-275a4a0e40b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cc6b31f-npwbj\" (UID: \"becfdd7d-d2f1-4a08-aeca-275a4a0e40b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc6b31f-npwbj" Oct 30 23:23:55.443406 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:55.443382 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74r4m\" (UniqueName: \"kubernetes.io/projected/becfdd7d-d2f1-4a08-aeca-275a4a0e40b2-kube-api-access-74r4m\") pod \"maas-api-revoke-sub-0cc6b31f-npwbj\" (UID: \"becfdd7d-d2f1-4a08-aeca-275a4a0e40b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc6b31f-npwbj" Oct 30 23:23:55.488816 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:55.488787 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb"] Oct 30 23:23:55.490162 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:23:55.490139 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee837874_e627_482a_8b34_19c8c5b55fc5.slice/crio-c19abf1dd8fb76c704805f6048759f06fa0e8f7a779deae6d5c1bda49caf5768 WatchSource:0}: Error finding container c19abf1dd8fb76c704805f6048759f06fa0e8f7a779deae6d5c1bda49caf5768: Status 404 returned error can't find the container with id c19abf1dd8fb76c704805f6048759f06fa0e8f7a779deae6d5c1bda49caf5768 Oct 30 23:23:55.492800 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:55.492764 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:23:55.492890 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:55.492834 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:23:55.492890 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:55.492864 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:23:55.523049 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:55.522997 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc6b31f-npwbj" Oct 30 23:23:55.910808 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:55.910780 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cc6b31f-npwbj"] Oct 30 23:23:55.912109 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:23:55.912080 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbecfdd7d_d2f1_4a08_aeca_275a4a0e40b2.slice/crio-d2aab80d4879a15d5dfb7f32c4eac52800477ad72852aa41a9fd214304b2bbe8 WatchSource:0}: Error finding container d2aab80d4879a15d5dfb7f32c4eac52800477ad72852aa41a9fd214304b2bbe8: Status 404 returned error can't find the container with id d2aab80d4879a15d5dfb7f32c4eac52800477ad72852aa41a9fd214304b2bbe8 Oct 30 23:23:55.914275 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:55.914256 4952 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 23:23:56.236968 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:56.236852 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" event={"ID":"ee837874-e627-482a-8b34-19c8c5b55fc5","Type":"ContainerStarted","Data":"048135fd65777bc3a84bdc765207f9e0ab6c430e3b478b8bdb224499745e7915"} Oct 30 23:23:56.236968 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:56.236916 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" event={"ID":"ee837874-e627-482a-8b34-19c8c5b55fc5","Type":"ContainerStarted","Data":"c19abf1dd8fb76c704805f6048759f06fa0e8f7a779deae6d5c1bda49caf5768"} Oct 30 23:23:56.238177 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:56.238155 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc6b31f-npwbj" event={"ID":"becfdd7d-d2f1-4a08-aeca-275a4a0e40b2","Type":"ContainerStarted","Data":"d2aab80d4879a15d5dfb7f32c4eac52800477ad72852aa41a9fd214304b2bbe8"} Oct 30 23:23:56.357242 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:56.357171 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" podStartSLOduration=2.357145755 podStartE2EDuration="2.357145755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:23:56.353774968 +0000 UTC m=+1055.497605173" watchObservedRunningTime="2026-10-30 23:23:56.357145755 +0000 UTC m=+1055.500975961" Oct 30 23:23:57.064994 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:57.064968 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" Oct 30 23:23:57.089068 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:57.088150 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-6zvgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 23:23:57.089068 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:57.088210 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" podUID="ee837874-e627-482a-8b34-19c8c5b55fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 23:23:57.705674 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:57.705635 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:23:58.064873 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:58.064797 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-6zvgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 23:23:58.064873 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:58.064851 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" podUID="ee837874-e627-482a-8b34-19c8c5b55fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 23:23:59.090065 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:59.089148 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-6zvgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 23:23:59.090065 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:23:59.089215 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" podUID="ee837874-e627-482a-8b34-19c8c5b55fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 23:24:00.064572 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:00.064542 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-6zvgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 23:24:00.064727 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:00.064595 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" podUID="ee837874-e627-482a-8b34-19c8c5b55fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 23:24:01.064860 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:01.064830 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-6zvgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 23:24:01.065343 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:01.064881 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" podUID="ee837874-e627-482a-8b34-19c8c5b55fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 23:24:01.295119 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:01.295065 4952 generic.go:356] "Generic (PLEG): container finished" podID="becfdd7d-d2f1-4a08-aeca-275a4a0e40b2" containerID="5918aa8982e370c1213a5cdd870149a91a1c42e01a1e6fa073d33277ad275092" exitCode=0 Oct 30 23:24:01.295279 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:01.295187 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc6b31f-npwbj" event={"ID":"becfdd7d-d2f1-4a08-aeca-275a4a0e40b2","Type":"ContainerDied","Data":"5918aa8982e370c1213a5cdd870149a91a1c42e01a1e6fa073d33277ad275092"} Oct 30 23:24:02.065892 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:02.065373 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-6zvgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 23:24:02.065892 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:02.065429 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" podUID="ee837874-e627-482a-8b34-19c8c5b55fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 23:24:02.470240 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:02.470212 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc6b31f-npwbj" Oct 30 23:24:02.518268 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:02.518227 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:24:02.575869 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:02.575838 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/becfdd7d-d2f1-4a08-aeca-275a4a0e40b2-kube-api-access-74r4m\" (UniqueName: \"kubernetes.io/projected/becfdd7d-d2f1-4a08-aeca-275a4a0e40b2-kube-api-access-74r4m\") pod \"becfdd7d-d2f1-4a08-aeca-275a4a0e40b2\" (UID: \"becfdd7d-d2f1-4a08-aeca-275a4a0e40b2\") " Oct 30 23:24:02.576032 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:02.575905 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/becfdd7d-d2f1-4a08-aeca-275a4a0e40b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/becfdd7d-d2f1-4a08-aeca-275a4a0e40b2-maas-api-xks-ca\") pod \"becfdd7d-d2f1-4a08-aeca-275a4a0e40b2\" (UID: \"becfdd7d-d2f1-4a08-aeca-275a4a0e40b2\") " Oct 30 23:24:02.576032 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:02.575952 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/becfdd7d-d2f1-4a08-aeca-275a4a0e40b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/becfdd7d-d2f1-4a08-aeca-275a4a0e40b2-maas-api-openshift-ca\") pod \"becfdd7d-d2f1-4a08-aeca-275a4a0e40b2\" (UID: \"becfdd7d-d2f1-4a08-aeca-275a4a0e40b2\") " Oct 30 23:24:02.576340 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:02.576313 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/becfdd7d-d2f1-4a08-aeca-275a4a0e40b2-maas-api-openshift-ca" pod "becfdd7d-d2f1-4a08-aeca-275a4a0e40b2" (UID: "becfdd7d-d2f1-4a08-aeca-275a4a0e40b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:24:02.578264 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:02.578239 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/becfdd7d-d2f1-4a08-aeca-275a4a0e40b2-maas-api-xks-ca" pod "becfdd7d-d2f1-4a08-aeca-275a4a0e40b2" (UID: "becfdd7d-d2f1-4a08-aeca-275a4a0e40b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:24:02.578362 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:02.578336 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/becfdd7d-d2f1-4a08-aeca-275a4a0e40b2-kube-api-access-74r4m" pod "becfdd7d-d2f1-4a08-aeca-275a4a0e40b2" (UID: "becfdd7d-d2f1-4a08-aeca-275a4a0e40b2"). InnerVolumeSpecName "kube-api-access-74r4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:24:02.677824 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:02.677735 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74r4m\" (UniqueName: \"kubernetes.io/projected/becfdd7d-d2f1-4a08-aeca-275a4a0e40b2-kube-api-access-74r4m\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:24:02.677824 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:02.677769 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/becfdd7d-d2f1-4a08-aeca-275a4a0e40b2-maas-api-xks-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:24:02.677824 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:02.677785 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/becfdd7d-d2f1-4a08-aeca-275a4a0e40b2-maas-api-openshift-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:24:03.064946 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:03.064914 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-6zvgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 23:24:03.065129 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:03.064968 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" podUID="ee837874-e627-482a-8b34-19c8c5b55fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 23:24:03.324017 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:03.323913 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc6b31f-npwbj" event={"ID":"becfdd7d-d2f1-4a08-aeca-275a4a0e40b2","Type":"ContainerDied","Data":"d2aab80d4879a15d5dfb7f32c4eac52800477ad72852aa41a9fd214304b2bbe8"} Oct 30 23:24:03.324017 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:03.323988 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2aab80d4879a15d5dfb7f32c4eac52800477ad72852aa41a9fd214304b2bbe8" Oct 30 23:24:03.324017 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:03.323929 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc6b31f-npwbj" Oct 30 23:24:04.099808 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:04.099777 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" Oct 30 23:24:04.100360 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:04.100311 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" Oct 30 23:24:04.101612 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:04.101590 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" Oct 30 23:24:07.332800 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:07.332768 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:24:07.652931 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:07.652903 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:24:09.874721 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:09.874684 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t"] Oct 30 23:24:09.875366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:09.875347 4952 cpu_manager.go:414] "containerMap: removing container" podUID="becfdd7d-d2f1-4a08-aeca-275a4a0e40b2" containerName="revoke-keys" Oct 30 23:24:09.875460 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:09.875367 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="becfdd7d-d2f1-4a08-aeca-275a4a0e40b2" containerName="revoke-keys" Oct 30 23:24:09.875582 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:09.875541 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="becfdd7d-d2f1-4a08-aeca-275a4a0e40b2" containerName="revoke-keys" Oct 30 23:24:09.878920 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:09.878898 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" Oct 30 23:24:09.958788 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:09.958757 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21849d16-2991-4d48-b38c-a3557eeaa97e-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-69s4t\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" Oct 30 23:24:09.958912 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:09.958815 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21849d16-2991-4d48-b38c-a3557eeaa97e-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-69s4t\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" Oct 30 23:24:09.958962 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:09.958934 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21849d16-2991-4d48-b38c-a3557eeaa97e-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-69s4t\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" Oct 30 23:24:09.959002 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:09.958988 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27ntl\" (UniqueName: \"kubernetes.io/projected/21849d16-2991-4d48-b38c-a3557eeaa97e-kube-api-access-27ntl\") pod \"maas-default-gateway-openshift-default-5df48fcc99-69s4t\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" Oct 30 23:24:09.959137 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:09.959117 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21849d16-2991-4d48-b38c-a3557eeaa97e-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-69s4t\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" Oct 30 23:24:09.959189 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:09.959173 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21849d16-2991-4d48-b38c-a3557eeaa97e-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-69s4t\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" Oct 30 23:24:09.959282 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:09.959258 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21849d16-2991-4d48-b38c-a3557eeaa97e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-69s4t\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" Oct 30 23:24:09.959396 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:09.959377 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21849d16-2991-4d48-b38c-a3557eeaa97e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-69s4t\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" Oct 30 23:24:09.959467 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:09.959451 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21849d16-2991-4d48-b38c-a3557eeaa97e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-69s4t\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" Oct 30 23:24:10.060403 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:10.060369 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21849d16-2991-4d48-b38c-a3557eeaa97e-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-69s4t\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" Oct 30 23:24:10.060403 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:10.060405 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21849d16-2991-4d48-b38c-a3557eeaa97e-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-69s4t\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" Oct 30 23:24:10.060635 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:10.060428 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21849d16-2991-4d48-b38c-a3557eeaa97e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-69s4t\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" Oct 30 23:24:10.060635 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:10.060477 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21849d16-2991-4d48-b38c-a3557eeaa97e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-69s4t\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" Oct 30 23:24:10.060635 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:10.060507 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21849d16-2991-4d48-b38c-a3557eeaa97e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-69s4t\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" Oct 30 23:24:10.060635 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:10.060526 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21849d16-2991-4d48-b38c-a3557eeaa97e-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-69s4t\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" Oct 30 23:24:10.060635 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:10.060549 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21849d16-2991-4d48-b38c-a3557eeaa97e-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-69s4t\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" Oct 30 23:24:10.060635 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:10.060602 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21849d16-2991-4d48-b38c-a3557eeaa97e-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-69s4t\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" Oct 30 23:24:10.060945 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:10.060693 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27ntl\" (UniqueName: \"kubernetes.io/projected/21849d16-2991-4d48-b38c-a3557eeaa97e-kube-api-access-27ntl\") pod \"maas-default-gateway-openshift-default-5df48fcc99-69s4t\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" Oct 30 23:24:10.060945 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:10.060894 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21849d16-2991-4d48-b38c-a3557eeaa97e-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-69s4t\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" Oct 30 23:24:10.061078 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:10.061054 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21849d16-2991-4d48-b38c-a3557eeaa97e-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-69s4t\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" Oct 30 23:24:10.061159 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:10.061097 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21849d16-2991-4d48-b38c-a3557eeaa97e-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-69s4t\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" Oct 30 23:24:10.061223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:10.061170 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21849d16-2991-4d48-b38c-a3557eeaa97e-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-69s4t\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" Oct 30 23:24:10.061472 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:10.061449 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21849d16-2991-4d48-b38c-a3557eeaa97e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-69s4t\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" Oct 30 23:24:10.063645 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:10.063620 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21849d16-2991-4d48-b38c-a3557eeaa97e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-69s4t\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" Oct 30 23:24:10.063833 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:10.063815 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21849d16-2991-4d48-b38c-a3557eeaa97e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-69s4t\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" Oct 30 23:24:10.075646 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:10.075613 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t"] Oct 30 23:24:10.118323 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:10.118297 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21849d16-2991-4d48-b38c-a3557eeaa97e-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-69s4t\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" Oct 30 23:24:10.118607 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:10.118583 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27ntl\" (UniqueName: \"kubernetes.io/projected/21849d16-2991-4d48-b38c-a3557eeaa97e-kube-api-access-27ntl\") pod \"maas-default-gateway-openshift-default-5df48fcc99-69s4t\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" Oct 30 23:24:10.191987 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:10.191913 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" Oct 30 23:24:10.620731 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:10.620702 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t"] Oct 30 23:24:10.622200 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:24:10.622177 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21849d16_2991_4d48_b38c_a3557eeaa97e.slice/crio-9fcddd1832d28a82eee9afa4bc5c43498ddc99087096059d9e63fd8c774f95b0 WatchSource:0}: Error finding container 9fcddd1832d28a82eee9afa4bc5c43498ddc99087096059d9e63fd8c774f95b0: Status 404 returned error can't find the container with id 9fcddd1832d28a82eee9afa4bc5c43498ddc99087096059d9e63fd8c774f95b0 Oct 30 23:24:10.624577 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:10.624541 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:24:10.624652 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:10.624607 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:24:10.624652 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:10.624634 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:24:11.098623 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:11.098594 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:24:11.381705 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:11.381603 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" event={"ID":"21849d16-2991-4d48-b38c-a3557eeaa97e","Type":"ContainerStarted","Data":"8ce37109a3a5d66f090a76470b8a76a809304109a30e8ab85074b80d1e8c7955"} Oct 30 23:24:11.383149 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:11.383120 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" event={"ID":"21849d16-2991-4d48-b38c-a3557eeaa97e","Type":"ContainerStarted","Data":"9fcddd1832d28a82eee9afa4bc5c43498ddc99087096059d9e63fd8c774f95b0"} Oct 30 23:24:12.196246 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.196208 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" Oct 30 23:24:12.197971 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.197943 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-69s4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 23:24:12.198239 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.198207 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" podUID="21849d16-2991-4d48-b38c-a3557eeaa97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 23:24:12.207120 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.207098 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:24:12.587119 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.584726 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" podStartSLOduration=3.584705422 podStartE2EDuration="3.584705422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:24:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:24:11.602865837 +0000 UTC m=+1070.746696079" watchObservedRunningTime="2026-10-30 23:24:12.584705422 +0000 UTC m=+1071.728535627" Oct 30 23:24:12.591215 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.591183 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs"] Oct 30 23:24:12.596098 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.596078 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" Oct 30 23:24:12.664843 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.664813 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs"] Oct 30 23:24:12.704456 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.704426 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-istio-podinfo\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" Oct 30 23:24:12.704619 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.704480 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-istio-envoy\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" Oct 30 23:24:12.704619 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.704547 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-workload-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" Oct 30 23:24:12.704744 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.704664 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k4mx\" (UniqueName: \"kubernetes.io/projected/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-kube-api-access-8k4mx\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" Oct 30 23:24:12.704744 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.704701 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-workload-certs\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" Oct 30 23:24:12.704873 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.704857 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-credential-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" Oct 30 23:24:12.704935 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.704910 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-istio-data\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" Oct 30 23:24:12.704977 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.704963 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-istiod-ca-cert\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" Oct 30 23:24:12.705026 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.705005 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-istio-token\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" Oct 30 23:24:12.720503 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.720481 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-3c9613-openshift-default-dockercfg-n6hzr\"" Oct 30 23:24:12.806302 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.806277 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-credential-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" Oct 30 23:24:12.806436 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.806318 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-istio-data\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" Oct 30 23:24:12.806436 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.806343 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-istiod-ca-cert\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" Oct 30 23:24:12.806436 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.806371 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-istio-token\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" Oct 30 23:24:12.806436 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.806390 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-istio-podinfo\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" Oct 30 23:24:12.806436 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.806413 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-istio-envoy\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" Oct 30 23:24:12.806436 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.806429 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-workload-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" Oct 30 23:24:12.806743 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.806478 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k4mx\" (UniqueName: \"kubernetes.io/projected/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-kube-api-access-8k4mx\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" Oct 30 23:24:12.806743 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.806515 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-workload-certs\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" Oct 30 23:24:12.806849 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.806737 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-credential-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" Oct 30 23:24:12.806849 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.806808 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-istio-data\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" Oct 30 23:24:12.806960 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.806905 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-workload-certs\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" Oct 30 23:24:12.807145 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.807114 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-workload-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" Oct 30 23:24:12.807145 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.807141 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-istiod-ca-cert\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" Oct 30 23:24:12.809048 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.809011 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-istio-envoy\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" Oct 30 23:24:12.809135 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.809102 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-istio-podinfo\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" Oct 30 23:24:12.863908 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.863849 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-istio-token\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" Oct 30 23:24:12.864063 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.864019 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k4mx\" (UniqueName: \"kubernetes.io/projected/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-kube-api-access-8k4mx\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" Oct 30 23:24:12.910257 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:12.910232 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" Oct 30 23:24:13.193136 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:13.193060 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-69s4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 23:24:13.193136 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:13.193119 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" podUID="21849d16-2991-4d48-b38c-a3557eeaa97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 23:24:13.390064 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:13.389235 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs"] Oct 30 23:24:13.405325 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:24:13.405303 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7e392e_f85e_4e50_8b1c_c0423a05a38c.slice/crio-53e0aa872dde246fe012602e7e91c8b670889f6358ef48d4c57a39a4aea4e195 WatchSource:0}: Error finding container 53e0aa872dde246fe012602e7e91c8b670889f6358ef48d4c57a39a4aea4e195: Status 404 returned error can't find the container with id 53e0aa872dde246fe012602e7e91c8b670889f6358ef48d4c57a39a4aea4e195 Oct 30 23:24:13.407587 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:13.407559 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:24:13.407662 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:13.407624 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:24:13.407662 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:13.407653 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:24:14.199558 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:14.199339 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-69s4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 23:24:14.199558 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:14.199397 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" podUID="21849d16-2991-4d48-b38c-a3557eeaa97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 23:24:14.419181 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:14.419139 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" event={"ID":"bf7e392e-f85e-4e50-8b1c-c0423a05a38c","Type":"ContainerStarted","Data":"8fab35f8572f4b75affb68e6d7bf7ac7724dea345535f431c8d6e30543c43f2d"} Oct 30 23:24:14.419814 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:14.419771 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" event={"ID":"bf7e392e-f85e-4e50-8b1c-c0423a05a38c","Type":"ContainerStarted","Data":"53e0aa872dde246fe012602e7e91c8b670889f6358ef48d4c57a39a4aea4e195"} Oct 30 23:24:14.552594 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:14.552490 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" podStartSLOduration=2.5524768460000002 podStartE2EDuration="2.552476846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:24:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:24:14.550308047 +0000 UTC m=+1073.694138249" watchObservedRunningTime="2026-10-30 23:24:14.552476846 +0000 UTC m=+1073.696307050" Oct 30 23:24:14.913189 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:14.913149 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" Oct 30 23:24:14.913493 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:14.913472 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 23:24:14.913675 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:14.913652 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" podUID="bf7e392e-f85e-4e50-8b1c-c0423a05a38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 23:24:15.193526 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:15.193429 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-69s4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 23:24:15.193799 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:15.193771 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" podUID="21849d16-2991-4d48-b38c-a3557eeaa97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 23:24:15.390076 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:15.390022 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:24:15.617077 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:15.616906 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:24:15.916721 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:15.916616 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 23:24:15.917014 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:15.916967 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" podUID="bf7e392e-f85e-4e50-8b1c-c0423a05a38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 23:24:16.193483 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:16.193396 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-69s4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 23:24:16.193738 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:16.193711 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" podUID="21849d16-2991-4d48-b38c-a3557eeaa97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 23:24:16.539050 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:16.538953 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:24:16.911780 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:16.911739 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 23:24:16.912335 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:16.911811 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" podUID="bf7e392e-f85e-4e50-8b1c-c0423a05a38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 23:24:17.193493 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:17.193404 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-69s4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 23:24:17.193760 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:17.193734 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" podUID="21849d16-2991-4d48-b38c-a3557eeaa97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 23:24:17.326638 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:17.326601 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:24:17.911542 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:17.911510 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 23:24:17.911808 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:17.911783 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" podUID="bf7e392e-f85e-4e50-8b1c-c0423a05a38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 23:24:18.193594 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:18.193114 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-69s4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 23:24:18.193594 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:18.193183 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" podUID="21849d16-2991-4d48-b38c-a3557eeaa97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 23:24:18.911949 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:18.911913 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 23:24:18.912565 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:18.912532 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" podUID="bf7e392e-f85e-4e50-8b1c-c0423a05a38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 23:24:19.706371 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:19.706339 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" Oct 30 23:24:19.707243 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:19.707219 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" Oct 30 23:24:19.708524 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:19.708496 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" Oct 30 23:24:19.911149 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:19.911107 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 23:24:19.911301 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:19.911179 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" podUID="bf7e392e-f85e-4e50-8b1c-c0423a05a38c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 23:24:21.146317 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:24:21.146279 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:24:21.462075 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:21.461945 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" Oct 30 23:24:21.462433 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:21.462400 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" Oct 30 23:24:21.464080 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:21.464062 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" Oct 30 23:24:22.276689 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:22.276660 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:24:27.091473 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:27.091433 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:24:27.283683 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:27.283650 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:24:31.877100 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:31.877047 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:24:36.794216 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:36.794174 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:24:41.542396 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.542363 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq"] Oct 30 23:24:41.546312 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.546289 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" Oct 30 23:24:41.609029 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.609004 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-5dc3c0-openshift-default-dockercfg-f5r9h\"" Oct 30 23:24:41.633653 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.633626 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/293e802c-8e13-4e98-9026-8f13ee239f58-workload-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" Oct 30 23:24:41.633771 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.633681 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/293e802c-8e13-4e98-9026-8f13ee239f58-istio-data\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" Oct 30 23:24:41.633813 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.633767 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/293e802c-8e13-4e98-9026-8f13ee239f58-istio-podinfo\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" Oct 30 23:24:41.633847 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.633820 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/293e802c-8e13-4e98-9026-8f13ee239f58-istiod-ca-cert\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" Oct 30 23:24:41.633887 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.633852 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/293e802c-8e13-4e98-9026-8f13ee239f58-istio-envoy\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" Oct 30 23:24:41.633920 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.633900 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4rqp\" (UniqueName: \"kubernetes.io/projected/293e802c-8e13-4e98-9026-8f13ee239f58-kube-api-access-m4rqp\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" Oct 30 23:24:41.633953 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.633942 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/293e802c-8e13-4e98-9026-8f13ee239f58-credential-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" Oct 30 23:24:41.634060 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.634018 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/293e802c-8e13-4e98-9026-8f13ee239f58-istio-token\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" Oct 30 23:24:41.634154 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.634138 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/293e802c-8e13-4e98-9026-8f13ee239f58-workload-certs\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" Oct 30 23:24:41.657062 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.657005 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq"] Oct 30 23:24:41.665015 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.664979 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:24:41.735393 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.735366 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/293e802c-8e13-4e98-9026-8f13ee239f58-workload-certs\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" Oct 30 23:24:41.735526 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.735412 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/293e802c-8e13-4e98-9026-8f13ee239f58-workload-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" Oct 30 23:24:41.735526 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.735471 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/293e802c-8e13-4e98-9026-8f13ee239f58-istio-data\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" Oct 30 23:24:41.735526 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.735514 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/293e802c-8e13-4e98-9026-8f13ee239f58-istio-podinfo\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" Oct 30 23:24:41.735655 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.735535 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/293e802c-8e13-4e98-9026-8f13ee239f58-istiod-ca-cert\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" Oct 30 23:24:41.735655 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.735562 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/293e802c-8e13-4e98-9026-8f13ee239f58-istio-envoy\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" Oct 30 23:24:41.735655 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.735606 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4rqp\" (UniqueName: \"kubernetes.io/projected/293e802c-8e13-4e98-9026-8f13ee239f58-kube-api-access-m4rqp\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" Oct 30 23:24:41.735800 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.735668 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/293e802c-8e13-4e98-9026-8f13ee239f58-credential-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" Oct 30 23:24:41.735850 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.735832 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/293e802c-8e13-4e98-9026-8f13ee239f58-workload-certs\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" Oct 30 23:24:41.735903 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.735867 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/293e802c-8e13-4e98-9026-8f13ee239f58-workload-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" Oct 30 23:24:41.735959 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.735947 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/293e802c-8e13-4e98-9026-8f13ee239f58-istio-token\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" Oct 30 23:24:41.736013 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.735980 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/293e802c-8e13-4e98-9026-8f13ee239f58-istio-data\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" Oct 30 23:24:41.736210 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.736192 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/293e802c-8e13-4e98-9026-8f13ee239f58-credential-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" Oct 30 23:24:41.736503 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.736333 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/293e802c-8e13-4e98-9026-8f13ee239f58-istiod-ca-cert\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" Oct 30 23:24:41.737961 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.737942 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/293e802c-8e13-4e98-9026-8f13ee239f58-istio-envoy\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" Oct 30 23:24:41.738326 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.738306 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/293e802c-8e13-4e98-9026-8f13ee239f58-istio-podinfo\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" Oct 30 23:24:41.792564 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.792510 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/293e802c-8e13-4e98-9026-8f13ee239f58-istio-token\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" Oct 30 23:24:41.792757 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.792736 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4rqp\" (UniqueName: \"kubernetes.io/projected/293e802c-8e13-4e98-9026-8f13ee239f58-kube-api-access-m4rqp\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" Oct 30 23:24:41.841491 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.841467 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w"] Oct 30 23:24:41.846597 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.846578 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" Oct 30 23:24:41.861381 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.861359 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" Oct 30 23:24:41.902551 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.902514 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-aadaa6-openshift-default-dockercfg-6bsq5\"" Oct 30 23:24:41.906558 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.906532 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w"] Oct 30 23:24:41.939012 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.938605 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmxw7\" (UniqueName: \"kubernetes.io/projected/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-kube-api-access-cmxw7\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" Oct 30 23:24:41.939012 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.938667 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-workload-certs\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" Oct 30 23:24:41.939012 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.938683 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-istio-envoy\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" Oct 30 23:24:41.939012 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.938743 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-istio-podinfo\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" Oct 30 23:24:41.939012 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.938778 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-istio-data\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" Oct 30 23:24:41.939012 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.938799 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-workload-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" Oct 30 23:24:41.939012 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.938824 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-istio-token\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" Oct 30 23:24:41.939012 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.938846 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-credential-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" Oct 30 23:24:41.939012 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:41.938871 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-istiod-ca-cert\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" Oct 30 23:24:42.040123 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:42.040093 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-istio-data\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" Oct 30 23:24:42.040284 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:42.040128 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-workload-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" Oct 30 23:24:42.040284 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:42.040157 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-istio-token\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" Oct 30 23:24:42.040284 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:42.040179 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-credential-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" Oct 30 23:24:42.040284 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:42.040205 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-istiod-ca-cert\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" Oct 30 23:24:42.040284 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:42.040240 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmxw7\" (UniqueName: \"kubernetes.io/projected/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-kube-api-access-cmxw7\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" Oct 30 23:24:42.040540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:42.040288 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-workload-certs\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" Oct 30 23:24:42.040540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:42.040313 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-istio-envoy\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" Oct 30 23:24:42.040540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:42.040391 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-istio-podinfo\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" Oct 30 23:24:42.040689 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:42.040580 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-workload-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" Oct 30 23:24:42.040689 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:42.040623 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-istio-data\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" Oct 30 23:24:42.040689 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:42.040681 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-credential-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" Oct 30 23:24:42.040817 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:42.040797 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-workload-certs\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" Oct 30 23:24:42.040911 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:42.040895 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-istiod-ca-cert\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" Oct 30 23:24:42.042784 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:42.042741 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-istio-envoy\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" Oct 30 23:24:42.043009 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:42.042993 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-istio-podinfo\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" Oct 30 23:24:42.098919 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:42.098894 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-istio-token\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" Oct 30 23:24:42.099026 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:42.098987 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmxw7\" (UniqueName: \"kubernetes.io/projected/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-kube-api-access-cmxw7\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" Oct 30 23:24:42.160623 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:42.160599 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" Oct 30 23:24:42.291585 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:42.291557 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq"] Oct 30 23:24:42.292590 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:24:42.292561 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod293e802c_8e13_4e98_9026_8f13ee239f58.slice/crio-645fc1133f50bed09dc6453407def5cbbd62aa3c4e990b440969f90a011dad73 WatchSource:0}: Error finding container 645fc1133f50bed09dc6453407def5cbbd62aa3c4e990b440969f90a011dad73: Status 404 returned error can't find the container with id 645fc1133f50bed09dc6453407def5cbbd62aa3c4e990b440969f90a011dad73 Oct 30 23:24:42.295771 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:42.295726 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:24:42.295902 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:42.295878 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:24:42.295991 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:42.295970 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:24:42.547763 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:42.547735 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w"] Oct 30 23:24:42.548168 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:24:42.547908 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eb85bfe_fe64_4db7_abc4_6a453d8c3060.slice/crio-80a1e028e8a5462c5430a679b9f8dd35b172220b2ca3ae68564e0fa3c1a59e11 WatchSource:0}: Error finding container 80a1e028e8a5462c5430a679b9f8dd35b172220b2ca3ae68564e0fa3c1a59e11: Status 404 returned error can't find the container with id 80a1e028e8a5462c5430a679b9f8dd35b172220b2ca3ae68564e0fa3c1a59e11 Oct 30 23:24:42.550410 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:42.550376 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:24:42.550516 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:42.550442 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:24:42.550516 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:42.550470 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:24:42.612655 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:42.612625 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" event={"ID":"293e802c-8e13-4e98-9026-8f13ee239f58","Type":"ContainerStarted","Data":"c0d6e0408408a3d44f7780b7f4ed0083b45b700ed3b191b16ce20395de32ca13"} Oct 30 23:24:42.612775 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:42.612684 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" event={"ID":"293e802c-8e13-4e98-9026-8f13ee239f58","Type":"ContainerStarted","Data":"645fc1133f50bed09dc6453407def5cbbd62aa3c4e990b440969f90a011dad73"} Oct 30 23:24:42.614817 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:42.614795 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" event={"ID":"9eb85bfe-fe64-4db7-abc4-6a453d8c3060","Type":"ContainerStarted","Data":"80a1e028e8a5462c5430a679b9f8dd35b172220b2ca3ae68564e0fa3c1a59e11"} Oct 30 23:24:42.729620 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:42.729559 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" podStartSLOduration=1.729541082 podStartE2EDuration="1.729541082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:24:42.728664919 +0000 UTC m=+1101.872495126" watchObservedRunningTime="2026-10-30 23:24:42.729541082 +0000 UTC m=+1101.873371288" Oct 30 23:24:42.862329 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:42.862301 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" Oct 30 23:24:43.631605 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:43.631571 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" event={"ID":"9eb85bfe-fe64-4db7-abc4-6a453d8c3060","Type":"ContainerStarted","Data":"6212e45b4f25274285b0b8bf55c23a7bb4ac32dd5f61ad03d9d361cbd54a61d8"} Oct 30 23:24:43.815405 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:43.815329 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" podStartSLOduration=2.815308214 podStartE2EDuration="2.815308214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:24:43.799620576 +0000 UTC m=+1102.943450782" watchObservedRunningTime="2026-10-30 23:24:43.815308214 +0000 UTC m=+1102.959138420" Oct 30 23:24:43.863697 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:43.863662 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 23:24:43.863855 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:43.863710 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" podUID="293e802c-8e13-4e98-9026-8f13ee239f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 23:24:44.138149 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:44.138121 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:24:44.161635 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:44.161612 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" Oct 30 23:24:44.162942 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:44.162923 4952 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 23:24:44.163015 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:44.162970 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" podUID="9eb85bfe-fe64-4db7-abc4-6a453d8c3060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 23:24:44.862112 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:44.862082 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 23:24:44.862522 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:44.862132 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" podUID="293e802c-8e13-4e98-9026-8f13ee239f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 23:24:45.160915 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:45.160838 4952 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 23:24:45.160915 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:45.160895 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" podUID="9eb85bfe-fe64-4db7-abc4-6a453d8c3060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 23:24:45.861636 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:45.861603 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 23:24:45.861817 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:45.861661 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" podUID="293e802c-8e13-4e98-9026-8f13ee239f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 23:24:46.160985 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:46.160904 4952 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 23:24:46.161478 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:46.161446 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" podUID="9eb85bfe-fe64-4db7-abc4-6a453d8c3060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 23:24:46.653655 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:46.653611 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:24:46.862091 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:46.862063 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 23:24:46.862290 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:46.862267 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" podUID="293e802c-8e13-4e98-9026-8f13ee239f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 23:24:47.161423 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:47.161388 4952 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 23:24:47.161801 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:47.161445 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" podUID="9eb85bfe-fe64-4db7-abc4-6a453d8c3060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 23:24:47.862360 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:47.862330 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 23:24:47.862693 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:47.862671 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" podUID="293e802c-8e13-4e98-9026-8f13ee239f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 23:24:48.161383 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:48.161301 4952 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 23:24:48.161383 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:48.161347 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" podUID="9eb85bfe-fe64-4db7-abc4-6a453d8c3060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 23:24:48.646012 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:48.645975 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz"] Oct 30 23:24:48.652029 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:48.652001 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" Oct 30 23:24:48.707919 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:48.707885 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz"] Oct 30 23:24:48.749800 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:48.749763 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-989a4f\"/\"distinct-w4-989a4f-kserve-self-signed-certs\"" Oct 30 23:24:48.757390 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:48.757362 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28913940-90f7-4d80-89b7-f5eece975d02-tmp-dir\") pod \"distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz\" (UID: \"28913940-90f7-4d80-89b7-f5eece975d02\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" Oct 30 23:24:48.757492 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:48.757440 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28913940-90f7-4d80-89b7-f5eece975d02-tls-certs\") pod \"distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz\" (UID: \"28913940-90f7-4d80-89b7-f5eece975d02\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" Oct 30 23:24:48.757492 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:48.757485 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px427\" (UniqueName: \"kubernetes.io/projected/28913940-90f7-4d80-89b7-f5eece975d02-kube-api-access-px427\") pod \"distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz\" (UID: \"28913940-90f7-4d80-89b7-f5eece975d02\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" Oct 30 23:24:48.757599 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:48.757507 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28913940-90f7-4d80-89b7-f5eece975d02-model-cache\") pod \"distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz\" (UID: \"28913940-90f7-4d80-89b7-f5eece975d02\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" Oct 30 23:24:48.757689 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:48.757666 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28913940-90f7-4d80-89b7-f5eece975d02-dshm\") pod \"distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz\" (UID: \"28913940-90f7-4d80-89b7-f5eece975d02\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" Oct 30 23:24:48.757750 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:48.757731 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28913940-90f7-4d80-89b7-f5eece975d02-home\") pod \"distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz\" (UID: \"28913940-90f7-4d80-89b7-f5eece975d02\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" Oct 30 23:24:48.816608 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:48.816575 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:24:48.858360 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:48.858332 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28913940-90f7-4d80-89b7-f5eece975d02-model-cache\") pod \"distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz\" (UID: \"28913940-90f7-4d80-89b7-f5eece975d02\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" Oct 30 23:24:48.858544 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:48.858386 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28913940-90f7-4d80-89b7-f5eece975d02-dshm\") pod \"distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz\" (UID: \"28913940-90f7-4d80-89b7-f5eece975d02\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" Oct 30 23:24:48.858544 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:48.858426 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28913940-90f7-4d80-89b7-f5eece975d02-home\") pod \"distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz\" (UID: \"28913940-90f7-4d80-89b7-f5eece975d02\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" Oct 30 23:24:48.858669 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:48.858557 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28913940-90f7-4d80-89b7-f5eece975d02-tmp-dir\") pod \"distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz\" (UID: \"28913940-90f7-4d80-89b7-f5eece975d02\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" Oct 30 23:24:48.858669 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:48.858595 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28913940-90f7-4d80-89b7-f5eece975d02-tls-certs\") pod \"distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz\" (UID: \"28913940-90f7-4d80-89b7-f5eece975d02\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" Oct 30 23:24:48.858771 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:48.858678 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px427\" (UniqueName: \"kubernetes.io/projected/28913940-90f7-4d80-89b7-f5eece975d02-kube-api-access-px427\") pod \"distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz\" (UID: \"28913940-90f7-4d80-89b7-f5eece975d02\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" Oct 30 23:24:48.858821 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:48.858800 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28913940-90f7-4d80-89b7-f5eece975d02-model-cache\") pod \"distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz\" (UID: \"28913940-90f7-4d80-89b7-f5eece975d02\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" Oct 30 23:24:48.859198 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:48.859173 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28913940-90f7-4d80-89b7-f5eece975d02-tmp-dir\") pod \"distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz\" (UID: \"28913940-90f7-4d80-89b7-f5eece975d02\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" Oct 30 23:24:48.859455 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:48.859432 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28913940-90f7-4d80-89b7-f5eece975d02-home\") pod \"distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz\" (UID: \"28913940-90f7-4d80-89b7-f5eece975d02\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" Oct 30 23:24:48.861902 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:48.861876 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28913940-90f7-4d80-89b7-f5eece975d02-tls-certs\") pod \"distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz\" (UID: \"28913940-90f7-4d80-89b7-f5eece975d02\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" Oct 30 23:24:48.862795 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:48.862389 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 23:24:48.862795 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:48.862450 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" podUID="293e802c-8e13-4e98-9026-8f13ee239f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 23:24:48.862980 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:48.862908 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28913940-90f7-4d80-89b7-f5eece975d02-dshm\") pod \"distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz\" (UID: \"28913940-90f7-4d80-89b7-f5eece975d02\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" Oct 30 23:24:48.919176 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:48.919093 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px427\" (UniqueName: \"kubernetes.io/projected/28913940-90f7-4d80-89b7-f5eece975d02-kube-api-access-px427\") pod \"distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz\" (UID: \"28913940-90f7-4d80-89b7-f5eece975d02\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" Oct 30 23:24:48.966139 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:48.966006 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" Oct 30 23:24:49.161648 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:49.161178 4952 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 23:24:49.161648 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:49.161241 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" podUID="9eb85bfe-fe64-4db7-abc4-6a453d8c3060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 23:24:49.433755 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:49.433727 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz"] Oct 30 23:24:49.435872 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:24:49.435847 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28913940_90f7_4d80_89b7_f5eece975d02.slice/crio-3e66f279c6ffdb315bd26749946400c4cf4a49a1d6e75370c8be757c6d656c89 WatchSource:0}: Error finding container 3e66f279c6ffdb315bd26749946400c4cf4a49a1d6e75370c8be757c6d656c89: Status 404 returned error can't find the container with id 3e66f279c6ffdb315bd26749946400c4cf4a49a1d6e75370c8be757c6d656c89 Oct 30 23:24:49.683309 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:49.683208 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" event={"ID":"28913940-90f7-4d80-89b7-f5eece975d02","Type":"ContainerStarted","Data":"56504e0ec32c81c45b4e864516ca85c7c433d760801756c25a038e67a15606ba"} Oct 30 23:24:49.683309 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:49.683260 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" event={"ID":"28913940-90f7-4d80-89b7-f5eece975d02","Type":"ContainerStarted","Data":"3e66f279c6ffdb315bd26749946400c4cf4a49a1d6e75370c8be757c6d656c89"} Oct 30 23:24:49.799881 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:49.799833 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" podStartSLOduration=1.7998183349999999 podStartE2EDuration="1.799818335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:24:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:24:49.796191904 +0000 UTC m=+1108.940022109" watchObservedRunningTime="2026-10-30 23:24:49.799818335 +0000 UTC m=+1108.943648534" Oct 30 23:24:49.866995 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:49.866968 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" Oct 30 23:24:49.867502 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:49.867478 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" Oct 30 23:24:49.868318 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:49.868298 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" Oct 30 23:24:49.966746 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:49.966657 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" Oct 30 23:24:50.166001 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:50.165974 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" Oct 30 23:24:50.166494 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:50.166473 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" Oct 30 23:24:50.167193 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:50.167169 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" Oct 30 23:24:51.142238 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:24:51.142201 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:24:51.576084 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:51.575973 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:24:53.186990 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:53.186952 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:24:56.363203 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:56.363160 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:24:58.970924 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:58.970893 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" Oct 30 23:24:58.980600 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:58.980574 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" Oct 30 23:24:59.761705 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:24:59.761675 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" Oct 30 23:25:01.159092 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:01.159053 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:25:05.921479 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:05.921438 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:25:06.253921 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:06.253843 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj"] Oct 30 23:25:06.257405 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:06.257388 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" Oct 30 23:25:06.351304 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:06.351279 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab5286a5-6020-4c0b-9511-db63d4472a0a-tmp-dir\") pod \"distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj\" (UID: \"ab5286a5-6020-4c0b-9511-db63d4472a0a\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" Oct 30 23:25:06.351455 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:06.351336 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab5286a5-6020-4c0b-9511-db63d4472a0a-dshm\") pod \"distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj\" (UID: \"ab5286a5-6020-4c0b-9511-db63d4472a0a\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" Oct 30 23:25:06.351455 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:06.351410 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab5286a5-6020-4c0b-9511-db63d4472a0a-home\") pod \"distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj\" (UID: \"ab5286a5-6020-4c0b-9511-db63d4472a0a\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" Oct 30 23:25:06.351533 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:06.351472 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab5286a5-6020-4c0b-9511-db63d4472a0a-model-cache\") pod \"distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj\" (UID: \"ab5286a5-6020-4c0b-9511-db63d4472a0a\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" Oct 30 23:25:06.351585 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:06.351562 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab5286a5-6020-4c0b-9511-db63d4472a0a-tls-certs\") pod \"distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj\" (UID: \"ab5286a5-6020-4c0b-9511-db63d4472a0a\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" Oct 30 23:25:06.351632 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:06.351619 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvsk2\" (UniqueName: \"kubernetes.io/projected/ab5286a5-6020-4c0b-9511-db63d4472a0a-kube-api-access-jvsk2\") pod \"distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj\" (UID: \"ab5286a5-6020-4c0b-9511-db63d4472a0a\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" Oct 30 23:25:06.360474 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:06.360446 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b81a7a\"/\"distinct-w1-b81a7a-kserve-self-signed-certs\"" Oct 30 23:25:06.363658 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:06.363641 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj"] Oct 30 23:25:06.433353 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:06.433319 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:25:06.452627 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:06.452595 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab5286a5-6020-4c0b-9511-db63d4472a0a-model-cache\") pod \"distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj\" (UID: \"ab5286a5-6020-4c0b-9511-db63d4472a0a\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" Oct 30 23:25:06.452851 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:06.452669 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab5286a5-6020-4c0b-9511-db63d4472a0a-tls-certs\") pod \"distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj\" (UID: \"ab5286a5-6020-4c0b-9511-db63d4472a0a\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" Oct 30 23:25:06.452851 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:06.452718 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvsk2\" (UniqueName: \"kubernetes.io/projected/ab5286a5-6020-4c0b-9511-db63d4472a0a-kube-api-access-jvsk2\") pod \"distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj\" (UID: \"ab5286a5-6020-4c0b-9511-db63d4472a0a\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" Oct 30 23:25:06.452851 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:06.452808 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab5286a5-6020-4c0b-9511-db63d4472a0a-tmp-dir\") pod \"distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj\" (UID: \"ab5286a5-6020-4c0b-9511-db63d4472a0a\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" Oct 30 23:25:06.453160 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:06.452892 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab5286a5-6020-4c0b-9511-db63d4472a0a-dshm\") pod \"distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj\" (UID: \"ab5286a5-6020-4c0b-9511-db63d4472a0a\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" Oct 30 23:25:06.453160 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:06.452936 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab5286a5-6020-4c0b-9511-db63d4472a0a-home\") pod \"distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj\" (UID: \"ab5286a5-6020-4c0b-9511-db63d4472a0a\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" Oct 30 23:25:06.453160 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:06.453135 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab5286a5-6020-4c0b-9511-db63d4472a0a-model-cache\") pod \"distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj\" (UID: \"ab5286a5-6020-4c0b-9511-db63d4472a0a\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" Oct 30 23:25:06.453348 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:06.453328 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab5286a5-6020-4c0b-9511-db63d4472a0a-home\") pod \"distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj\" (UID: \"ab5286a5-6020-4c0b-9511-db63d4472a0a\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" Oct 30 23:25:06.453627 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:06.453608 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab5286a5-6020-4c0b-9511-db63d4472a0a-tmp-dir\") pod \"distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj\" (UID: \"ab5286a5-6020-4c0b-9511-db63d4472a0a\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" Oct 30 23:25:06.456209 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:06.456184 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab5286a5-6020-4c0b-9511-db63d4472a0a-tls-certs\") pod \"distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj\" (UID: \"ab5286a5-6020-4c0b-9511-db63d4472a0a\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" Oct 30 23:25:06.456209 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:06.456200 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab5286a5-6020-4c0b-9511-db63d4472a0a-dshm\") pod \"distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj\" (UID: \"ab5286a5-6020-4c0b-9511-db63d4472a0a\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" Oct 30 23:25:06.511088 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:06.511006 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvsk2\" (UniqueName: \"kubernetes.io/projected/ab5286a5-6020-4c0b-9511-db63d4472a0a-kube-api-access-jvsk2\") pod \"distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj\" (UID: \"ab5286a5-6020-4c0b-9511-db63d4472a0a\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" Oct 30 23:25:06.570933 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:06.570902 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" Oct 30 23:25:06.981407 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:06.981374 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj"] Oct 30 23:25:06.981884 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:25:06.981775 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab5286a5_6020_4c0b_9511_db63d4472a0a.slice/crio-a4c6070e1cf580aa66ea2590181191ced825c2ef3ebf33048a2bb5641576fd5c WatchSource:0}: Error finding container a4c6070e1cf580aa66ea2590181191ced825c2ef3ebf33048a2bb5641576fd5c: Status 404 returned error can't find the container with id a4c6070e1cf580aa66ea2590181191ced825c2ef3ebf33048a2bb5641576fd5c Oct 30 23:25:07.808679 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:07.808635 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" event={"ID":"ab5286a5-6020-4c0b-9511-db63d4472a0a","Type":"ContainerStarted","Data":"6072ad05f62dd188fd1a506af9929ba3000f2bc3cd3bac0fe3c40e33cbd73c84"} Oct 30 23:25:07.808875 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:07.808684 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" event={"ID":"ab5286a5-6020-4c0b-9511-db63d4472a0a","Type":"ContainerStarted","Data":"a4c6070e1cf580aa66ea2590181191ced825c2ef3ebf33048a2bb5641576fd5c"} Oct 30 23:25:07.923950 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:07.923905 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" podStartSLOduration=1.9238927700000001 podStartE2EDuration="1.92389277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:25:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:25:07.922205677 +0000 UTC m=+1127.066035883" watchObservedRunningTime="2026-10-30 23:25:07.92389277 +0000 UTC m=+1127.067722975" Oct 30 23:25:08.571351 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:08.571321 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" Oct 30 23:25:10.668318 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:10.668244 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:25:12.424536 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:12.424506 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr"] Oct 30 23:25:12.428471 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:12.428446 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" Oct 30 23:25:12.515737 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:12.515704 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ca32053-b157-4808-b0e8-c7a2c02a58b9-istio-token\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" Oct 30 23:25:12.515877 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:12.515788 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca32053-b157-4808-b0e8-c7a2c02a58b9-workload-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" Oct 30 23:25:12.515877 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:12.515830 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca32053-b157-4808-b0e8-c7a2c02a58b9-credential-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" Oct 30 23:25:12.516058 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:12.515911 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ca32053-b157-4808-b0e8-c7a2c02a58b9-istio-podinfo\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" Oct 30 23:25:12.516109 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:12.516075 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dkzc\" (UniqueName: \"kubernetes.io/projected/1ca32053-b157-4808-b0e8-c7a2c02a58b9-kube-api-access-7dkzc\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" Oct 30 23:25:12.516160 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:12.516125 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ca32053-b157-4808-b0e8-c7a2c02a58b9-istiod-ca-cert\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" Oct 30 23:25:12.516228 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:12.516210 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ca32053-b157-4808-b0e8-c7a2c02a58b9-istio-envoy\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" Oct 30 23:25:12.516289 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:12.516273 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ca32053-b157-4808-b0e8-c7a2c02a58b9-istio-data\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" Oct 30 23:25:12.516341 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:12.516313 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ca32053-b157-4808-b0e8-c7a2c02a58b9-workload-certs\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" Oct 30 23:25:12.617185 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:12.617146 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ca32053-b157-4808-b0e8-c7a2c02a58b9-istio-data\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" Oct 30 23:25:12.617740 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:12.617198 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ca32053-b157-4808-b0e8-c7a2c02a58b9-workload-certs\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" Oct 30 23:25:12.617740 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:12.617244 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ca32053-b157-4808-b0e8-c7a2c02a58b9-istio-token\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" Oct 30 23:25:12.617740 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:12.617274 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca32053-b157-4808-b0e8-c7a2c02a58b9-workload-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" Oct 30 23:25:12.617740 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:12.617300 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca32053-b157-4808-b0e8-c7a2c02a58b9-credential-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" Oct 30 23:25:12.617740 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:12.617350 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ca32053-b157-4808-b0e8-c7a2c02a58b9-istio-podinfo\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" Oct 30 23:25:12.617740 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:12.617412 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dkzc\" (UniqueName: \"kubernetes.io/projected/1ca32053-b157-4808-b0e8-c7a2c02a58b9-kube-api-access-7dkzc\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" Oct 30 23:25:12.617740 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:12.617447 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ca32053-b157-4808-b0e8-c7a2c02a58b9-istiod-ca-cert\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" Oct 30 23:25:12.617740 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:12.617502 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ca32053-b157-4808-b0e8-c7a2c02a58b9-istio-envoy\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" Oct 30 23:25:12.617740 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:12.617588 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ca32053-b157-4808-b0e8-c7a2c02a58b9-istio-data\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" Oct 30 23:25:12.617740 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:12.617605 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ca32053-b157-4808-b0e8-c7a2c02a58b9-workload-certs\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" Oct 30 23:25:12.617740 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:12.617683 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca32053-b157-4808-b0e8-c7a2c02a58b9-credential-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" Oct 30 23:25:12.618238 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:12.617768 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca32053-b157-4808-b0e8-c7a2c02a58b9-workload-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" Oct 30 23:25:12.618277 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:12.618255 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ca32053-b157-4808-b0e8-c7a2c02a58b9-istiod-ca-cert\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" Oct 30 23:25:12.620056 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:12.620022 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ca32053-b157-4808-b0e8-c7a2c02a58b9-istio-envoy\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" Oct 30 23:25:12.620303 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:12.620283 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ca32053-b157-4808-b0e8-c7a2c02a58b9-istio-podinfo\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" Oct 30 23:25:12.638507 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:12.638482 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr"] Oct 30 23:25:12.676060 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:12.675979 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ca32053-b157-4808-b0e8-c7a2c02a58b9-istio-token\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" Oct 30 23:25:12.676401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:12.676380 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dkzc\" (UniqueName: \"kubernetes.io/projected/1ca32053-b157-4808-b0e8-c7a2c02a58b9-kube-api-access-7dkzc\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" Oct 30 23:25:12.741908 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:12.741883 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" Oct 30 23:25:13.172750 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:13.172723 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr"] Oct 30 23:25:13.173430 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:25:13.173397 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca32053_b157_4808_b0e8_c7a2c02a58b9.slice/crio-78dd3fde84bee9a72f474c9a9d6c1e512c108c9c05b20a793f8284e01da110d0 WatchSource:0}: Error finding container 78dd3fde84bee9a72f474c9a9d6c1e512c108c9c05b20a793f8284e01da110d0: Status 404 returned error can't find the container with id 78dd3fde84bee9a72f474c9a9d6c1e512c108c9c05b20a793f8284e01da110d0 Oct 30 23:25:13.175735 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:13.175701 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:25:13.175817 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:13.175767 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:25:13.175817 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:13.175797 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:25:13.850513 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:13.850481 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" event={"ID":"1ca32053-b157-4808-b0e8-c7a2c02a58b9","Type":"ContainerStarted","Data":"edcf8dfcafe5bed2ae29dd5e87cdf6854338b0a1fe273ac492a037affb170921"} Oct 30 23:25:13.850956 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:13.850540 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" event={"ID":"1ca32053-b157-4808-b0e8-c7a2c02a58b9","Type":"ContainerStarted","Data":"78dd3fde84bee9a72f474c9a9d6c1e512c108c9c05b20a793f8284e01da110d0"} Oct 30 23:25:14.742394 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:14.742360 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" Oct 30 23:25:14.744343 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:14.744301 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 23:25:14.744461 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:14.744365 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" podUID="1ca32053-b157-4808-b0e8-c7a2c02a58b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 23:25:15.608622 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:15.608583 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:25:15.742540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:15.742509 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 23:25:15.742704 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:15.742586 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" podUID="1ca32053-b157-4808-b0e8-c7a2c02a58b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 23:25:16.591216 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:16.588420 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" Oct 30 23:25:16.605227 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:16.605200 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" Oct 30 23:25:16.720486 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:16.720434 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" podStartSLOduration=4.720419747 podStartE2EDuration="4.720419747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:25:14.069285499 +0000 UTC m=+1133.213115752" watchObservedRunningTime="2026-10-30 23:25:16.720419747 +0000 UTC m=+1135.864249952" Oct 30 23:25:16.742678 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:16.742648 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 23:25:16.742812 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:16.742719 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" podUID="1ca32053-b157-4808-b0e8-c7a2c02a58b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 23:25:16.901206 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:16.901109 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:25:16.906512 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:16.906485 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" Oct 30 23:25:17.742350 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:17.742317 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 23:25:17.744425 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:17.742375 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" podUID="1ca32053-b157-4808-b0e8-c7a2c02a58b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 23:25:18.742853 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:18.742818 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 23:25:18.743278 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:18.742867 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" podUID="1ca32053-b157-4808-b0e8-c7a2c02a58b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 23:25:19.354815 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:19.354777 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:25:19.743084 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:19.742968 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 23:25:19.743476 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:19.743077 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" podUID="1ca32053-b157-4808-b0e8-c7a2c02a58b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 23:25:20.475655 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:20.475615 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:25:20.747224 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:20.747147 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" Oct 30 23:25:20.747640 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:20.747620 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" Oct 30 23:25:20.748438 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:20.748414 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" Oct 30 23:25:20.955444 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:20.955409 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:25:21.153261 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:25:21.153216 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:25:24.698658 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:24.698604 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:25:25.287494 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:25.287455 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:25:27.847247 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:27.847216 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6"] Oct 30 23:25:27.852292 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:27.852274 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" Oct 30 23:25:27.904854 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:27.904829 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-989a4f-openshift-default-dockercfg-qv7mz\"" Oct 30 23:25:27.922627 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:27.922602 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac8fce7-3841-4fb4-a7a3-06359477953b-workload-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" Oct 30 23:25:27.922736 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:27.922635 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ac8fce7-3841-4fb4-a7a3-06359477953b-istio-envoy\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" Oct 30 23:25:27.922736 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:27.922653 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ac8fce7-3841-4fb4-a7a3-06359477953b-istio-token\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" Oct 30 23:25:27.922813 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:27.922733 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ac8fce7-3841-4fb4-a7a3-06359477953b-istio-podinfo\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" Oct 30 23:25:27.922813 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:27.922761 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac8fce7-3841-4fb4-a7a3-06359477953b-credential-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" Oct 30 23:25:27.922875 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:27.922815 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ac8fce7-3841-4fb4-a7a3-06359477953b-workload-certs\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" Oct 30 23:25:27.922875 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:27.922861 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blndn\" (UniqueName: \"kubernetes.io/projected/7ac8fce7-3841-4fb4-a7a3-06359477953b-kube-api-access-blndn\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" Oct 30 23:25:27.922939 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:27.922922 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ac8fce7-3841-4fb4-a7a3-06359477953b-istiod-ca-cert\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" Oct 30 23:25:27.922972 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:27.922956 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ac8fce7-3841-4fb4-a7a3-06359477953b-istio-data\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" Oct 30 23:25:27.959574 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:27.959547 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6"] Oct 30 23:25:28.024261 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:28.024236 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac8fce7-3841-4fb4-a7a3-06359477953b-workload-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" Oct 30 23:25:28.024406 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:28.024268 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ac8fce7-3841-4fb4-a7a3-06359477953b-istio-envoy\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" Oct 30 23:25:28.024406 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:28.024286 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ac8fce7-3841-4fb4-a7a3-06359477953b-istio-token\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" Oct 30 23:25:28.024406 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:28.024315 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ac8fce7-3841-4fb4-a7a3-06359477953b-istio-podinfo\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" Oct 30 23:25:28.024406 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:28.024331 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac8fce7-3841-4fb4-a7a3-06359477953b-credential-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" Oct 30 23:25:28.024406 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:28.024351 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ac8fce7-3841-4fb4-a7a3-06359477953b-workload-certs\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" Oct 30 23:25:28.024643 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:28.024415 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blndn\" (UniqueName: \"kubernetes.io/projected/7ac8fce7-3841-4fb4-a7a3-06359477953b-kube-api-access-blndn\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" Oct 30 23:25:28.024643 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:28.024462 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ac8fce7-3841-4fb4-a7a3-06359477953b-istiod-ca-cert\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" Oct 30 23:25:28.024643 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:28.024528 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ac8fce7-3841-4fb4-a7a3-06359477953b-istio-data\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" Oct 30 23:25:28.025085 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:28.025057 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac8fce7-3841-4fb4-a7a3-06359477953b-workload-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" Oct 30 23:25:28.025232 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:28.025145 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac8fce7-3841-4fb4-a7a3-06359477953b-credential-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" Oct 30 23:25:28.025329 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:28.025250 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ac8fce7-3841-4fb4-a7a3-06359477953b-istiod-ca-cert\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" Oct 30 23:25:28.025329 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:28.025288 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ac8fce7-3841-4fb4-a7a3-06359477953b-workload-certs\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" Oct 30 23:25:28.025471 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:28.025447 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ac8fce7-3841-4fb4-a7a3-06359477953b-istio-data\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" Oct 30 23:25:28.027068 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:28.027002 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ac8fce7-3841-4fb4-a7a3-06359477953b-istio-podinfo\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" Oct 30 23:25:28.027200 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:28.027184 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ac8fce7-3841-4fb4-a7a3-06359477953b-istio-envoy\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" Oct 30 23:25:28.085239 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:28.085219 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ac8fce7-3841-4fb4-a7a3-06359477953b-istio-token\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" Oct 30 23:25:28.085444 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:28.085427 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blndn\" (UniqueName: \"kubernetes.io/projected/7ac8fce7-3841-4fb4-a7a3-06359477953b-kube-api-access-blndn\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" Oct 30 23:25:28.173861 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:28.173796 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" Oct 30 23:25:28.615085 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:25:28.615018 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ac8fce7_3841_4fb4_a7a3_06359477953b.slice/crio-3f2a0133acf8eae7d8a46cda3d3f75f72f700297e14e38c4270d999f2619070f WatchSource:0}: Error finding container 3f2a0133acf8eae7d8a46cda3d3f75f72f700297e14e38c4270d999f2619070f: Status 404 returned error can't find the container with id 3f2a0133acf8eae7d8a46cda3d3f75f72f700297e14e38c4270d999f2619070f Oct 30 23:25:28.616302 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:28.616273 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6"] Oct 30 23:25:28.617562 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:28.617527 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:25:28.617657 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:28.617595 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:25:28.617657 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:28.617627 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:25:28.987554 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:28.987466 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" event={"ID":"7ac8fce7-3841-4fb4-a7a3-06359477953b","Type":"ContainerStarted","Data":"f948ee65804062388c8493c7835ae278365527341d2d6f6d9f50f67c5936ba10"} Oct 30 23:25:28.987554 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:28.987513 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" event={"ID":"7ac8fce7-3841-4fb4-a7a3-06359477953b","Type":"ContainerStarted","Data":"3f2a0133acf8eae7d8a46cda3d3f75f72f700297e14e38c4270d999f2619070f"} Oct 30 23:25:29.106935 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:29.106876 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" podStartSLOduration=2.106858845 podStartE2EDuration="2.106858845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:25:29.105581135 +0000 UTC m=+1148.249411342" watchObservedRunningTime="2026-10-30 23:25:29.106858845 +0000 UTC m=+1148.250689124" Oct 30 23:25:29.180060 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:29.179106 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" Oct 30 23:25:29.180060 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:29.179156 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 23:25:29.180060 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:29.179201 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" podUID="7ac8fce7-3841-4fb4-a7a3-06359477953b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 23:25:30.132515 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:30.132473 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:25:30.177459 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:30.177430 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 23:25:30.177707 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:30.177682 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" podUID="7ac8fce7-3841-4fb4-a7a3-06359477953b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 23:25:30.747008 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:30.746975 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:25:31.179165 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:31.179131 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 23:25:31.179747 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:31.179715 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" podUID="7ac8fce7-3841-4fb4-a7a3-06359477953b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 23:25:32.184437 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:32.184396 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 23:25:32.185085 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:32.185020 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" podUID="7ac8fce7-3841-4fb4-a7a3-06359477953b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 23:25:33.175198 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:33.175162 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 23:25:33.175469 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:33.175440 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" podUID="7ac8fce7-3841-4fb4-a7a3-06359477953b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 23:25:34.184064 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:34.177371 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 23:25:34.184064 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:34.177441 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" podUID="7ac8fce7-3841-4fb4-a7a3-06359477953b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 23:25:35.023406 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:35.023368 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:25:35.175154 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:35.175123 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 23:25:35.175348 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:35.175179 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" podUID="7ac8fce7-3841-4fb4-a7a3-06359477953b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 23:25:36.178699 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:36.178669 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" Oct 30 23:25:36.179118 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:36.179068 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" Oct 30 23:25:36.179744 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:36.179723 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" Oct 30 23:25:39.821290 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:39.821257 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:25:43.805150 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:43.805109 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt"] Oct 30 23:25:43.809356 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:43.809336 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" Oct 30 23:25:43.865843 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:43.865814 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt"] Oct 30 23:25:43.911497 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:43.911468 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-3c9613\"/\"openshift-service-ca.crt\"" Oct 30 23:25:43.911497 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:43.911473 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-3c9613\"/\"kube-root-ca.crt\"" Oct 30 23:25:43.912672 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:43.912653 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-3c9613\"/\"default-dockercfg-w26f4\"" Oct 30 23:25:43.912717 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:43.912663 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-3c9613\"/\"distinct-w2-3c9613-kserve-self-signed-certs\"" Oct 30 23:25:43.935454 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:43.935427 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-tmp-dir\") pod \"distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt\" (UID: \"86a57a21-72bc-4ce4-8d1e-bf29214ad4de\") " pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" Oct 30 23:25:43.935568 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:43.935481 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmx65\" (UniqueName: \"kubernetes.io/projected/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-kube-api-access-kmx65\") pod \"distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt\" (UID: \"86a57a21-72bc-4ce4-8d1e-bf29214ad4de\") " pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" Oct 30 23:25:43.935608 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:43.935560 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-tls-certs\") pod \"distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt\" (UID: \"86a57a21-72bc-4ce4-8d1e-bf29214ad4de\") " pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" Oct 30 23:25:43.935660 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:43.935646 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-home\") pod \"distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt\" (UID: \"86a57a21-72bc-4ce4-8d1e-bf29214ad4de\") " pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" Oct 30 23:25:43.935767 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:43.935754 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-model-cache\") pod \"distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt\" (UID: \"86a57a21-72bc-4ce4-8d1e-bf29214ad4de\") " pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" Oct 30 23:25:43.935805 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:43.935778 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-dshm\") pod \"distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt\" (UID: \"86a57a21-72bc-4ce4-8d1e-bf29214ad4de\") " pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" Oct 30 23:25:43.985121 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:43.985016 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:25:44.037133 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:44.037097 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-home\") pod \"distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt\" (UID: \"86a57a21-72bc-4ce4-8d1e-bf29214ad4de\") " pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" Oct 30 23:25:44.037305 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:44.037199 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-model-cache\") pod \"distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt\" (UID: \"86a57a21-72bc-4ce4-8d1e-bf29214ad4de\") " pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" Oct 30 23:25:44.037305 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:44.037220 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-dshm\") pod \"distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt\" (UID: \"86a57a21-72bc-4ce4-8d1e-bf29214ad4de\") " pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" Oct 30 23:25:44.037305 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:44.037265 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-tmp-dir\") pod \"distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt\" (UID: \"86a57a21-72bc-4ce4-8d1e-bf29214ad4de\") " pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" Oct 30 23:25:44.037305 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:44.037301 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmx65\" (UniqueName: \"kubernetes.io/projected/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-kube-api-access-kmx65\") pod \"distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt\" (UID: \"86a57a21-72bc-4ce4-8d1e-bf29214ad4de\") " pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" Oct 30 23:25:44.037510 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:44.037348 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-tls-certs\") pod \"distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt\" (UID: \"86a57a21-72bc-4ce4-8d1e-bf29214ad4de\") " pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" Oct 30 23:25:44.037617 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:44.037577 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-home\") pod \"distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt\" (UID: \"86a57a21-72bc-4ce4-8d1e-bf29214ad4de\") " pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" Oct 30 23:25:44.037764 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:44.037599 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-model-cache\") pod \"distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt\" (UID: \"86a57a21-72bc-4ce4-8d1e-bf29214ad4de\") " pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" Oct 30 23:25:44.037764 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:44.037658 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-tmp-dir\") pod \"distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt\" (UID: \"86a57a21-72bc-4ce4-8d1e-bf29214ad4de\") " pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" Oct 30 23:25:44.039774 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:44.039750 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-dshm\") pod \"distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt\" (UID: \"86a57a21-72bc-4ce4-8d1e-bf29214ad4de\") " pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" Oct 30 23:25:44.040350 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:44.040330 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-tls-certs\") pod \"distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt\" (UID: \"86a57a21-72bc-4ce4-8d1e-bf29214ad4de\") " pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" Oct 30 23:25:44.094080 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:44.094050 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmx65\" (UniqueName: \"kubernetes.io/projected/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-kube-api-access-kmx65\") pod \"distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt\" (UID: \"86a57a21-72bc-4ce4-8d1e-bf29214ad4de\") " pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" Oct 30 23:25:44.121998 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:44.121971 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" Oct 30 23:25:44.573891 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:44.573864 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt"] Oct 30 23:25:44.575109 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:25:44.575087 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86a57a21_72bc_4ce4_8d1e_bf29214ad4de.slice/crio-a0598d54525aae44e083da581ebce15854b7743c496946c00f5952584ceee168 WatchSource:0}: Error finding container a0598d54525aae44e083da581ebce15854b7743c496946c00f5952584ceee168: Status 404 returned error can't find the container with id a0598d54525aae44e083da581ebce15854b7743c496946c00f5952584ceee168 Oct 30 23:25:44.630023 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:44.629994 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:25:45.122094 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:45.122054 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" event={"ID":"86a57a21-72bc-4ce4-8d1e-bf29214ad4de","Type":"ContainerStarted","Data":"96c59deefb3e7054f39ff5f6e62ce4ea81d772b539af4e5f248c5b06aa93b927"} Oct 30 23:25:45.122483 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:45.122103 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" event={"ID":"86a57a21-72bc-4ce4-8d1e-bf29214ad4de","Type":"ContainerStarted","Data":"a0598d54525aae44e083da581ebce15854b7743c496946c00f5952584ceee168"} Oct 30 23:25:45.122483 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:45.122155 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" Oct 30 23:25:45.237093 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:45.237023 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" podStartSLOduration=2.23700767 podStartE2EDuration="2.23700767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:25:45.235539292 +0000 UTC m=+1164.379369511" watchObservedRunningTime="2026-10-30 23:25:45.23700767 +0000 UTC m=+1164.380837875" Oct 30 23:25:49.251052 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:49.251014 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:25:49.491720 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:49.491683 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:25:51.143582 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:25:51.143549 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:25:54.122166 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:54.122130 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" Oct 30 23:25:54.134527 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:54.134497 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" Oct 30 23:25:54.195173 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:54.195143 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" Oct 30 23:25:54.378560 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:54.378470 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:25:56.721926 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:56.721884 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm"] Oct 30 23:25:56.727220 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:56.727196 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" Oct 30 23:25:56.779497 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:56.779472 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-057303-openshift-default-dockercfg-gz47w\"" Oct 30 23:25:56.784052 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:56.784014 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5889f46b-2aa6-4eab-a3e4-171b81b804de-istio-envoy\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" Oct 30 23:25:56.784166 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:56.784148 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5889f46b-2aa6-4eab-a3e4-171b81b804de-workload-socket\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" Oct 30 23:25:56.784219 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:56.784181 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5889f46b-2aa6-4eab-a3e4-171b81b804de-istiod-ca-cert\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" Oct 30 23:25:56.784266 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:56.784233 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82d84\" (UniqueName: \"kubernetes.io/projected/5889f46b-2aa6-4eab-a3e4-171b81b804de-kube-api-access-82d84\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" Oct 30 23:25:56.784327 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:56.784313 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5889f46b-2aa6-4eab-a3e4-171b81b804de-istio-token\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" Oct 30 23:25:56.784398 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:56.784386 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5889f46b-2aa6-4eab-a3e4-171b81b804de-workload-certs\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" Oct 30 23:25:56.784453 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:56.784409 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5889f46b-2aa6-4eab-a3e4-171b81b804de-istio-podinfo\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" Oct 30 23:25:56.784453 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:56.784427 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5889f46b-2aa6-4eab-a3e4-171b81b804de-credential-socket\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" Oct 30 23:25:56.784592 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:56.784572 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5889f46b-2aa6-4eab-a3e4-171b81b804de-istio-data\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" Oct 30 23:25:56.834515 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:56.834487 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm"] Oct 30 23:25:56.885460 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:56.885435 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82d84\" (UniqueName: \"kubernetes.io/projected/5889f46b-2aa6-4eab-a3e4-171b81b804de-kube-api-access-82d84\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" Oct 30 23:25:56.885581 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:56.885470 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5889f46b-2aa6-4eab-a3e4-171b81b804de-istio-token\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" Oct 30 23:25:56.885581 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:56.885507 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5889f46b-2aa6-4eab-a3e4-171b81b804de-workload-certs\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" Oct 30 23:25:56.885581 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:56.885528 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5889f46b-2aa6-4eab-a3e4-171b81b804de-istio-podinfo\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" Oct 30 23:25:56.885581 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:56.885546 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5889f46b-2aa6-4eab-a3e4-171b81b804de-credential-socket\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" Oct 30 23:25:56.885772 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:56.885584 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5889f46b-2aa6-4eab-a3e4-171b81b804de-istio-data\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" Oct 30 23:25:56.885772 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:56.885628 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5889f46b-2aa6-4eab-a3e4-171b81b804de-istio-envoy\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" Oct 30 23:25:56.885772 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:56.885709 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5889f46b-2aa6-4eab-a3e4-171b81b804de-workload-socket\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" Oct 30 23:25:56.885772 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:56.885739 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5889f46b-2aa6-4eab-a3e4-171b81b804de-istiod-ca-cert\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" Oct 30 23:25:56.886078 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:56.886056 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5889f46b-2aa6-4eab-a3e4-171b81b804de-credential-socket\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" Oct 30 23:25:56.886203 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:56.886175 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5889f46b-2aa6-4eab-a3e4-171b81b804de-workload-certs\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" Oct 30 23:25:56.886305 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:56.886207 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5889f46b-2aa6-4eab-a3e4-171b81b804de-istio-data\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" Oct 30 23:25:56.886305 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:56.886239 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5889f46b-2aa6-4eab-a3e4-171b81b804de-workload-socket\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" Oct 30 23:25:56.886836 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:56.886814 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5889f46b-2aa6-4eab-a3e4-171b81b804de-istiod-ca-cert\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" Oct 30 23:25:56.888227 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:56.888201 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5889f46b-2aa6-4eab-a3e4-171b81b804de-istio-envoy\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" Oct 30 23:25:56.888430 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:56.888412 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5889f46b-2aa6-4eab-a3e4-171b81b804de-istio-podinfo\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" Oct 30 23:25:56.942606 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:56.942577 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5889f46b-2aa6-4eab-a3e4-171b81b804de-istio-token\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" Oct 30 23:25:56.942723 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:56.942708 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82d84\" (UniqueName: \"kubernetes.io/projected/5889f46b-2aa6-4eab-a3e4-171b81b804de-kube-api-access-82d84\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" Oct 30 23:25:57.041756 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:57.041689 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" Oct 30 23:25:57.470166 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:25:57.470140 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5889f46b_2aa6_4eab_a3e4_171b81b804de.slice/crio-768a31c7d71f112216c921554ab5e581fc863033b145689d01f31d3473bec548 WatchSource:0}: Error finding container 768a31c7d71f112216c921554ab5e581fc863033b145689d01f31d3473bec548: Status 404 returned error can't find the container with id 768a31c7d71f112216c921554ab5e581fc863033b145689d01f31d3473bec548 Oct 30 23:25:57.470369 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:57.470349 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm"] Oct 30 23:25:57.472409 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:57.472377 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:25:57.472510 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:57.472437 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:25:57.472510 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:57.472465 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:25:58.215299 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:58.215268 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" event={"ID":"5889f46b-2aa6-4eab-a3e4-171b81b804de","Type":"ContainerStarted","Data":"a96c29f9b629337402ea621829c72e5a31b1e7ab2eb79ab1f496871cb288fc85"} Oct 30 23:25:58.215749 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:58.215320 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" event={"ID":"5889f46b-2aa6-4eab-a3e4-171b81b804de","Type":"ContainerStarted","Data":"768a31c7d71f112216c921554ab5e581fc863033b145689d01f31d3473bec548"} Oct 30 23:25:58.352766 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:58.352715 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" podStartSLOduration=2.352697325 podStartE2EDuration="2.352697325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:25:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:25:58.349018209 +0000 UTC m=+1177.492848411" watchObservedRunningTime="2026-10-30 23:25:58.352697325 +0000 UTC m=+1177.496527530" Oct 30 23:25:59.042343 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:59.042309 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" Oct 30 23:25:59.043907 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:59.043880 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 23:25:59.044066 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:59.043925 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" podUID="5889f46b-2aa6-4eab-a3e4-171b81b804de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 23:25:59.202644 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:25:59.202613 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:26:00.042888 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:00.042851 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 23:26:00.043345 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:00.042911 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" podUID="5889f46b-2aa6-4eab-a3e4-171b81b804de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 23:26:01.042461 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:01.042427 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 23:26:01.042663 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:01.042483 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" podUID="5889f46b-2aa6-4eab-a3e4-171b81b804de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 23:26:02.042511 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:02.042483 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 23:26:02.042983 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:02.042536 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" podUID="5889f46b-2aa6-4eab-a3e4-171b81b804de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 23:26:03.042507 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:03.042469 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 23:26:03.042678 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:03.042520 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" podUID="5889f46b-2aa6-4eab-a3e4-171b81b804de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 23:26:04.042489 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:04.042452 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 23:26:04.042718 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:04.042540 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" podUID="5889f46b-2aa6-4eab-a3e4-171b81b804de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 23:26:04.095883 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:04.095853 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:26:05.046502 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:05.046478 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" Oct 30 23:26:05.046902 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:05.046880 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" Oct 30 23:26:05.047619 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:05.047604 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" Oct 30 23:26:08.967624 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:08.967577 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:26:12.550604 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:12.550573 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg"] Oct 30 23:26:12.554668 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:12.554649 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" Oct 30 23:26:12.661361 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:12.661333 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg"] Oct 30 23:26:12.669927 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:12.669898 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/824cbd14-4a34-4df1-864e-63bb34900aab-workload-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" Oct 30 23:26:12.670111 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:12.669940 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/824cbd14-4a34-4df1-864e-63bb34900aab-workload-certs\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" Oct 30 23:26:12.670111 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:12.670095 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/824cbd14-4a34-4df1-864e-63bb34900aab-istio-podinfo\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" Oct 30 23:26:12.670340 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:12.670319 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/824cbd14-4a34-4df1-864e-63bb34900aab-istiod-ca-cert\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" Oct 30 23:26:12.670463 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:12.670437 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/824cbd14-4a34-4df1-864e-63bb34900aab-istio-data\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" Oct 30 23:26:12.670586 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:12.670569 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfp4z\" (UniqueName: \"kubernetes.io/projected/824cbd14-4a34-4df1-864e-63bb34900aab-kube-api-access-wfp4z\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" Oct 30 23:26:12.670640 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:12.670609 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/824cbd14-4a34-4df1-864e-63bb34900aab-credential-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" Oct 30 23:26:12.670640 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:12.670631 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/824cbd14-4a34-4df1-864e-63bb34900aab-istio-envoy\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" Oct 30 23:26:12.670710 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:12.670660 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/824cbd14-4a34-4df1-864e-63bb34900aab-istio-token\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" Oct 30 23:26:12.773002 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:12.772966 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/824cbd14-4a34-4df1-864e-63bb34900aab-istio-data\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" Oct 30 23:26:12.773330 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:12.773306 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfp4z\" (UniqueName: \"kubernetes.io/projected/824cbd14-4a34-4df1-864e-63bb34900aab-kube-api-access-wfp4z\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" Oct 30 23:26:12.775091 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:12.773470 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/824cbd14-4a34-4df1-864e-63bb34900aab-istio-data\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" Oct 30 23:26:12.775091 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:12.773484 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/824cbd14-4a34-4df1-864e-63bb34900aab-credential-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" Oct 30 23:26:12.775091 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:12.773549 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/824cbd14-4a34-4df1-864e-63bb34900aab-istio-envoy\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" Oct 30 23:26:12.775091 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:12.773602 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/824cbd14-4a34-4df1-864e-63bb34900aab-istio-token\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" Oct 30 23:26:12.775091 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:12.773711 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/824cbd14-4a34-4df1-864e-63bb34900aab-workload-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" Oct 30 23:26:12.775091 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:12.773741 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/824cbd14-4a34-4df1-864e-63bb34900aab-workload-certs\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" Oct 30 23:26:12.775091 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:12.773779 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/824cbd14-4a34-4df1-864e-63bb34900aab-istio-podinfo\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" Oct 30 23:26:12.775091 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:12.773847 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/824cbd14-4a34-4df1-864e-63bb34900aab-istiod-ca-cert\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" Oct 30 23:26:12.775091 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:12.774495 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/824cbd14-4a34-4df1-864e-63bb34900aab-credential-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" Oct 30 23:26:12.775091 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:12.774778 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/824cbd14-4a34-4df1-864e-63bb34900aab-workload-certs\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" Oct 30 23:26:12.782467 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:12.782382 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/824cbd14-4a34-4df1-864e-63bb34900aab-istiod-ca-cert\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" Oct 30 23:26:12.783758 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:12.780480 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/824cbd14-4a34-4df1-864e-63bb34900aab-workload-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" Oct 30 23:26:12.787297 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:12.787010 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/824cbd14-4a34-4df1-864e-63bb34900aab-istio-envoy\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" Oct 30 23:26:12.787297 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:12.787254 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/824cbd14-4a34-4df1-864e-63bb34900aab-istio-podinfo\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" Oct 30 23:26:12.830802 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:12.830746 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfp4z\" (UniqueName: \"kubernetes.io/projected/824cbd14-4a34-4df1-864e-63bb34900aab-kube-api-access-wfp4z\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" Oct 30 23:26:12.830802 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:12.830757 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/824cbd14-4a34-4df1-864e-63bb34900aab-istio-token\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" Oct 30 23:26:12.868828 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:12.868804 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" Oct 30 23:26:13.294258 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:13.294233 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg"] Oct 30 23:26:13.294962 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:26:13.294936 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod824cbd14_4a34_4df1_864e_63bb34900aab.slice/crio-db357e5d14434ea6504233e41bdc89ed176da3ed50a699bfce98349a9d5e41fa WatchSource:0}: Error finding container db357e5d14434ea6504233e41bdc89ed176da3ed50a699bfce98349a9d5e41fa: Status 404 returned error can't find the container with id db357e5d14434ea6504233e41bdc89ed176da3ed50a699bfce98349a9d5e41fa Oct 30 23:26:13.297247 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:13.297207 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:26:13.297354 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:13.297310 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:26:13.297420 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:13.297356 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:26:13.322951 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:13.322922 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" event={"ID":"824cbd14-4a34-4df1-864e-63bb34900aab","Type":"ContainerStarted","Data":"db357e5d14434ea6504233e41bdc89ed176da3ed50a699bfce98349a9d5e41fa"} Oct 30 23:26:13.769507 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:13.769471 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:26:14.331230 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:14.331195 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" event={"ID":"824cbd14-4a34-4df1-864e-63bb34900aab","Type":"ContainerStarted","Data":"18ad8f0ed81d0e16352354e96f03c6c0defa7d402c7c11102f4f0aaa88ee4b5d"} Oct 30 23:26:14.453203 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:14.453139 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" podStartSLOduration=2.45311996 podStartE2EDuration="2.45311996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:26:14.451246185 +0000 UTC m=+1193.595076389" watchObservedRunningTime="2026-10-30 23:26:14.45311996 +0000 UTC m=+1193.596950167" Oct 30 23:26:14.874467 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:14.871224 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" Oct 30 23:26:14.885993 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:14.885667 4952 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 23:26:14.885993 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:14.885728 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" podUID="824cbd14-4a34-4df1-864e-63bb34900aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 23:26:15.871684 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:15.871645 4952 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 23:26:15.871996 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:15.871703 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" podUID="824cbd14-4a34-4df1-864e-63bb34900aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 23:26:16.881070 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:16.876329 4952 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 23:26:16.881070 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:16.876386 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" podUID="824cbd14-4a34-4df1-864e-63bb34900aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 23:26:17.275584 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:17.275498 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:26:17.502535 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:17.502500 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:26:17.869381 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:17.869346 4952 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 23:26:17.869671 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:17.869638 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" podUID="824cbd14-4a34-4df1-864e-63bb34900aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 23:26:18.617461 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:18.617427 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:26:18.871448 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:18.871349 4952 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 23:26:18.877065 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:18.872641 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" podUID="824cbd14-4a34-4df1-864e-63bb34900aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 23:26:19.638912 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:19.638878 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:26:19.869077 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:19.869014 4952 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 23:26:19.869242 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:19.869093 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" podUID="824cbd14-4a34-4df1-864e-63bb34900aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 23:26:20.934585 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:20.934557 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" Oct 30 23:26:20.935098 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:20.935025 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" Oct 30 23:26:20.935784 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:20.935766 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" Oct 30 23:26:21.114874 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:21.114850 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft_91651ce2-4673-4ced-b26d-e94ff8829d11/extract/0.log" Oct 30 23:26:21.121670 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:21.121648 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft_91651ce2-4673-4ced-b26d-e94ff8829d11/extract/0.log" Oct 30 23:26:21.130611 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:21.130587 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/2.log" Oct 30 23:26:21.137031 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:21.137012 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/2.log" Oct 30 23:26:21.137189 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:21.137052 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 30 23:26:21.145970 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:21.145953 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 30 23:26:21.149502 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:26:21.149475 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:26:23.482326 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:23.482282 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:26:28.192025 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:28.191996 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:26:28.648779 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:28.648751 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2"] Oct 30 23:26:28.654224 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:28.654205 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" Oct 30 23:26:28.752020 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:28.751994 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2"] Oct 30 23:26:28.755330 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:28.755299 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98027a1c-59f6-4d79-a118-dae8ed53e50b-credential-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" Oct 30 23:26:28.755499 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:28.755454 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98027a1c-59f6-4d79-a118-dae8ed53e50b-istio-envoy\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" Oct 30 23:26:28.755578 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:28.755564 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98027a1c-59f6-4d79-a118-dae8ed53e50b-istio-data\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" Oct 30 23:26:28.755645 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:28.755599 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98027a1c-59f6-4d79-a118-dae8ed53e50b-istio-podinfo\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" Oct 30 23:26:28.755645 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:28.755634 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98027a1c-59f6-4d79-a118-dae8ed53e50b-workload-certs\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" Oct 30 23:26:28.755760 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:28.755672 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phs47\" (UniqueName: \"kubernetes.io/projected/98027a1c-59f6-4d79-a118-dae8ed53e50b-kube-api-access-phs47\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" Oct 30 23:26:28.755819 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:28.755771 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98027a1c-59f6-4d79-a118-dae8ed53e50b-workload-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" Oct 30 23:26:28.755875 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:28.755817 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98027a1c-59f6-4d79-a118-dae8ed53e50b-istiod-ca-cert\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" Oct 30 23:26:28.755941 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:28.755924 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98027a1c-59f6-4d79-a118-dae8ed53e50b-istio-token\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" Oct 30 23:26:28.856587 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:28.856556 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98027a1c-59f6-4d79-a118-dae8ed53e50b-istio-data\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" Oct 30 23:26:28.856587 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:28.856593 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98027a1c-59f6-4d79-a118-dae8ed53e50b-istio-podinfo\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" Oct 30 23:26:28.856811 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:28.856620 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98027a1c-59f6-4d79-a118-dae8ed53e50b-workload-certs\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" Oct 30 23:26:28.856811 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:28.856640 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phs47\" (UniqueName: \"kubernetes.io/projected/98027a1c-59f6-4d79-a118-dae8ed53e50b-kube-api-access-phs47\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" Oct 30 23:26:28.856811 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:28.856664 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98027a1c-59f6-4d79-a118-dae8ed53e50b-workload-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" Oct 30 23:26:28.856811 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:28.856695 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98027a1c-59f6-4d79-a118-dae8ed53e50b-istiod-ca-cert\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" Oct 30 23:26:28.856811 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:28.856755 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98027a1c-59f6-4d79-a118-dae8ed53e50b-istio-token\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" Oct 30 23:26:28.857114 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:28.856816 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98027a1c-59f6-4d79-a118-dae8ed53e50b-credential-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" Oct 30 23:26:28.857114 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:28.856884 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98027a1c-59f6-4d79-a118-dae8ed53e50b-istio-envoy\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" Oct 30 23:26:28.857114 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:28.857072 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98027a1c-59f6-4d79-a118-dae8ed53e50b-istio-data\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" Oct 30 23:26:28.857287 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:28.857137 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98027a1c-59f6-4d79-a118-dae8ed53e50b-workload-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" Oct 30 23:26:28.857287 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:28.857142 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98027a1c-59f6-4d79-a118-dae8ed53e50b-workload-certs\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" Oct 30 23:26:28.857442 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:28.857423 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98027a1c-59f6-4d79-a118-dae8ed53e50b-credential-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" Oct 30 23:26:28.857641 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:28.857619 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98027a1c-59f6-4d79-a118-dae8ed53e50b-istiod-ca-cert\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" Oct 30 23:26:28.859418 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:28.859395 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98027a1c-59f6-4d79-a118-dae8ed53e50b-istio-envoy\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" Oct 30 23:26:28.859486 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:28.859463 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98027a1c-59f6-4d79-a118-dae8ed53e50b-istio-podinfo\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" Oct 30 23:26:28.913704 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:28.913630 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98027a1c-59f6-4d79-a118-dae8ed53e50b-istio-token\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" Oct 30 23:26:28.913872 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:28.913698 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phs47\" (UniqueName: \"kubernetes.io/projected/98027a1c-59f6-4d79-a118-dae8ed53e50b-kube-api-access-phs47\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" Oct 30 23:26:28.968695 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:28.968671 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" Oct 30 23:26:29.399246 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:29.399221 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2"] Oct 30 23:26:29.399592 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:26:29.399504 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98027a1c_59f6_4d79_a118_dae8ed53e50b.slice/crio-1b27845d45342de99ca52ed8d7eb5b92ee8a8b0701276c4ba2ff8fa838d4e059 WatchSource:0}: Error finding container 1b27845d45342de99ca52ed8d7eb5b92ee8a8b0701276c4ba2ff8fa838d4e059: Status 404 returned error can't find the container with id 1b27845d45342de99ca52ed8d7eb5b92ee8a8b0701276c4ba2ff8fa838d4e059 Oct 30 23:26:29.401969 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:29.401938 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:26:29.402065 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:29.402013 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:26:29.402109 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:29.402089 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:26:29.433416 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:29.433391 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" event={"ID":"98027a1c-59f6-4d79-a118-dae8ed53e50b","Type":"ContainerStarted","Data":"1b27845d45342de99ca52ed8d7eb5b92ee8a8b0701276c4ba2ff8fa838d4e059"} Oct 30 23:26:30.444146 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:30.444058 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" event={"ID":"98027a1c-59f6-4d79-a118-dae8ed53e50b","Type":"ContainerStarted","Data":"a178f50b0fbcf4b2a7b8bf4b41a1c0b9750b7b8a9dd2a0b322ea02f8816d5c1e"} Oct 30 23:26:30.615851 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:30.615787 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" podStartSLOduration=2.6157675510000002 podStartE2EDuration="2.615767551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:26:30.613498212 +0000 UTC m=+1209.757328422" watchObservedRunningTime="2026-10-30 23:26:30.615767551 +0000 UTC m=+1209.759597760" Oct 30 23:26:30.991269 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:30.991231 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 23:26:30.991486 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:30.991314 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" podUID="98027a1c-59f6-4d79-a118-dae8ed53e50b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 23:26:30.997307 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:30.997280 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" Oct 30 23:26:31.981775 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:31.981357 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 23:26:31.981775 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:31.981413 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" podUID="98027a1c-59f6-4d79-a118-dae8ed53e50b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 23:26:32.983179 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:32.983142 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 23:26:32.983759 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:32.983728 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" podUID="98027a1c-59f6-4d79-a118-dae8ed53e50b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 23:26:33.263251 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:33.263174 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:26:33.990391 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:33.990360 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 23:26:33.990894 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:33.990508 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" podUID="98027a1c-59f6-4d79-a118-dae8ed53e50b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 23:26:34.989247 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:34.989213 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 23:26:34.989534 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:34.989507 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" podUID="98027a1c-59f6-4d79-a118-dae8ed53e50b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 23:26:35.969659 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:35.969618 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 23:26:35.970032 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:35.969685 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" podUID="98027a1c-59f6-4d79-a118-dae8ed53e50b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 23:26:36.974963 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:36.974931 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" Oct 30 23:26:36.975479 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:36.975407 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" Oct 30 23:26:36.976029 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:36.976011 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" Oct 30 23:26:38.071577 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:38.071527 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:26:38.383477 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:38.383425 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:26:42.992724 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:42.992694 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:26:47.760545 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:47.760492 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:26:51.142669 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:26:51.142626 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:26:52.006515 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:52.006479 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:26:52.570635 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:52.570603 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:26:57.574268 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:26:57.574213 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:27:02.372109 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:02.372075 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:27:03.173493 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:03.173455 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:27:07.285116 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:07.285082 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:27:12.306339 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:12.306298 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:27:12.540133 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:12.540104 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:27:13.369424 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:13.369392 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc"] Oct 30 23:27:13.373276 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:13.373256 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" Oct 30 23:27:13.495010 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:13.494979 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-istio-token\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" Oct 30 23:27:13.495010 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:13.495013 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-workload-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" Oct 30 23:27:13.495220 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:13.495148 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-credential-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" Oct 30 23:27:13.495220 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:13.495200 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-istiod-ca-cert\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" Oct 30 23:27:13.495318 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:13.495299 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-workload-certs\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" Oct 30 23:27:13.495354 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:13.495328 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-istio-envoy\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" Oct 30 23:27:13.495415 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:13.495404 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-istio-podinfo\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" Oct 30 23:27:13.495487 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:13.495471 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-istio-data\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" Oct 30 23:27:13.495524 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:13.495515 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5xq8\" (UniqueName: \"kubernetes.io/projected/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-kube-api-access-r5xq8\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" Oct 30 23:27:13.577811 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:13.577779 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc"] Oct 30 23:27:13.596911 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:13.596883 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-workload-certs\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" Oct 30 23:27:13.597090 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:13.596941 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-istio-envoy\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" Oct 30 23:27:13.597090 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:13.597022 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-istio-podinfo\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" Oct 30 23:27:13.597190 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:13.597091 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-istio-data\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" Oct 30 23:27:13.597190 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:13.597156 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5xq8\" (UniqueName: \"kubernetes.io/projected/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-kube-api-access-r5xq8\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" Oct 30 23:27:13.597263 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:13.597234 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-istio-token\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" Oct 30 23:27:13.597297 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:13.597260 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-workload-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" Oct 30 23:27:13.597353 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:13.597336 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-credential-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" Oct 30 23:27:13.597391 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:13.597367 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-workload-certs\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" Oct 30 23:27:13.597462 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:13.597442 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-istiod-ca-cert\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" Oct 30 23:27:13.597709 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:13.597684 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-credential-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" Oct 30 23:27:13.597935 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:13.597912 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-istio-data\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" Oct 30 23:27:13.598275 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:13.598251 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-workload-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" Oct 30 23:27:13.598637 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:13.598594 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-istiod-ca-cert\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" Oct 30 23:27:13.600158 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:13.600136 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-istio-envoy\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" Oct 30 23:27:13.600832 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:13.600816 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-istio-podinfo\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" Oct 30 23:27:13.655921 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:13.655855 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-istio-token\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" Oct 30 23:27:13.655921 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:13.655863 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5xq8\" (UniqueName: \"kubernetes.io/projected/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-kube-api-access-r5xq8\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" Oct 30 23:27:13.686970 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:13.686924 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" Oct 30 23:27:14.125091 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:27:14.125049 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0e935ae_8758_4f00_beca_d6b4c7c38dd6.slice/crio-168d2f53c0591b6ae572b9913fe2da9723da78dc470fd3951c91f299376904b7 WatchSource:0}: Error finding container 168d2f53c0591b6ae572b9913fe2da9723da78dc470fd3951c91f299376904b7: Status 404 returned error can't find the container with id 168d2f53c0591b6ae572b9913fe2da9723da78dc470fd3951c91f299376904b7 Oct 30 23:27:14.126713 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:14.126661 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc"] Oct 30 23:27:14.127607 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:14.127566 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:27:14.127708 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:14.127645 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:27:14.127708 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:14.127687 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:27:14.802329 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:14.802280 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" event={"ID":"c0e935ae-8758-4f00-beca-d6b4c7c38dd6","Type":"ContainerStarted","Data":"0bd97141d47ac3b35cd21541f673920f3ac09d5c0b7960dbc078ecff772775cc"} Oct 30 23:27:14.802706 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:14.802347 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" event={"ID":"c0e935ae-8758-4f00-beca-d6b4c7c38dd6","Type":"ContainerStarted","Data":"168d2f53c0591b6ae572b9913fe2da9723da78dc470fd3951c91f299376904b7"} Oct 30 23:27:15.690225 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:15.690190 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" Oct 30 23:27:15.692158 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:15.692121 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 23:27:15.692284 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:15.692179 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" podUID="c0e935ae-8758-4f00-beca-d6b4c7c38dd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 23:27:16.690454 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:16.690415 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 23:27:16.690942 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:16.690477 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" podUID="c0e935ae-8758-4f00-beca-d6b4c7c38dd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 23:27:17.207191 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:17.207159 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:27:17.616145 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:17.616116 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:27:17.690366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:17.690332 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 23:27:17.690563 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:17.690385 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" podUID="c0e935ae-8758-4f00-beca-d6b4c7c38dd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 23:27:18.690374 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:18.690336 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 23:27:18.690554 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:18.690400 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" podUID="c0e935ae-8758-4f00-beca-d6b4c7c38dd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 23:27:19.692060 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:19.691342 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 23:27:19.692060 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:19.691399 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" podUID="c0e935ae-8758-4f00-beca-d6b4c7c38dd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 23:27:20.689403 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:20.689360 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 23:27:20.689598 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:20.689432 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" podUID="c0e935ae-8758-4f00-beca-d6b4c7c38dd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 23:27:21.226066 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:27:21.225281 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:27:22.048743 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:22.048636 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" Oct 30 23:27:22.049912 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:22.049885 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" Oct 30 23:27:22.050932 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:22.050909 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" Oct 30 23:27:22.247059 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:22.246360 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" podStartSLOduration=9.246342763 podStartE2EDuration="9.246342763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:27:15.018990282 +0000 UTC m=+1254.162820487" watchObservedRunningTime="2026-10-30 23:27:22.246342763 +0000 UTC m=+1261.390172969" Oct 30 23:27:22.445063 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:22.444292 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:27:22.713659 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:22.713519 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:27:27.292881 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:27.292796 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:27:27.930388 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:27.930355 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:27:32.120357 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:32.120328 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:27:36.711399 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:36.711357 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj"] Oct 30 23:27:36.716707 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:36.716688 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" Oct 30 23:27:36.787782 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:36.787750 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-5dc3c0\"/\"distinct-2-w5-5dc3c0-kserve-self-signed-certs\"" Oct 30 23:27:36.788597 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:36.788571 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj"] Oct 30 23:27:36.842071 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:36.841993 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/709f007e-0ccc-40f5-816e-ed24702ac107-dshm\") pod \"distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj\" (UID: \"709f007e-0ccc-40f5-816e-ed24702ac107\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" Oct 30 23:27:36.842259 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:36.842230 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/709f007e-0ccc-40f5-816e-ed24702ac107-home\") pod \"distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj\" (UID: \"709f007e-0ccc-40f5-816e-ed24702ac107\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" Oct 30 23:27:36.842342 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:36.842314 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/709f007e-0ccc-40f5-816e-ed24702ac107-tmp-dir\") pod \"distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj\" (UID: \"709f007e-0ccc-40f5-816e-ed24702ac107\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" Oct 30 23:27:36.842399 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:36.842349 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxvhm\" (UniqueName: \"kubernetes.io/projected/709f007e-0ccc-40f5-816e-ed24702ac107-kube-api-access-zxvhm\") pod \"distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj\" (UID: \"709f007e-0ccc-40f5-816e-ed24702ac107\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" Oct 30 23:27:36.842453 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:36.842412 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/709f007e-0ccc-40f5-816e-ed24702ac107-model-cache\") pod \"distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj\" (UID: \"709f007e-0ccc-40f5-816e-ed24702ac107\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" Oct 30 23:27:36.842632 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:36.842603 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/709f007e-0ccc-40f5-816e-ed24702ac107-tls-certs\") pod \"distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj\" (UID: \"709f007e-0ccc-40f5-816e-ed24702ac107\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" Oct 30 23:27:36.880637 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:36.880602 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:27:36.944019 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:36.943978 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/709f007e-0ccc-40f5-816e-ed24702ac107-home\") pod \"distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj\" (UID: \"709f007e-0ccc-40f5-816e-ed24702ac107\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" Oct 30 23:27:36.944255 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:36.944174 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/709f007e-0ccc-40f5-816e-ed24702ac107-tmp-dir\") pod \"distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj\" (UID: \"709f007e-0ccc-40f5-816e-ed24702ac107\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" Oct 30 23:27:36.944255 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:36.944227 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxvhm\" (UniqueName: \"kubernetes.io/projected/709f007e-0ccc-40f5-816e-ed24702ac107-kube-api-access-zxvhm\") pod \"distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj\" (UID: \"709f007e-0ccc-40f5-816e-ed24702ac107\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" Oct 30 23:27:36.944378 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:36.944289 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/709f007e-0ccc-40f5-816e-ed24702ac107-model-cache\") pod \"distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj\" (UID: \"709f007e-0ccc-40f5-816e-ed24702ac107\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" Oct 30 23:27:36.944436 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:36.944390 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/709f007e-0ccc-40f5-816e-ed24702ac107-tls-certs\") pod \"distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj\" (UID: \"709f007e-0ccc-40f5-816e-ed24702ac107\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" Oct 30 23:27:36.944498 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:36.944455 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/709f007e-0ccc-40f5-816e-ed24702ac107-dshm\") pod \"distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj\" (UID: \"709f007e-0ccc-40f5-816e-ed24702ac107\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" Oct 30 23:27:36.944498 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:36.944483 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/709f007e-0ccc-40f5-816e-ed24702ac107-home\") pod \"distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj\" (UID: \"709f007e-0ccc-40f5-816e-ed24702ac107\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" Oct 30 23:27:36.944603 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:36.944505 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/709f007e-0ccc-40f5-816e-ed24702ac107-tmp-dir\") pod \"distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj\" (UID: \"709f007e-0ccc-40f5-816e-ed24702ac107\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" Oct 30 23:27:36.944832 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:36.944813 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/709f007e-0ccc-40f5-816e-ed24702ac107-model-cache\") pod \"distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj\" (UID: \"709f007e-0ccc-40f5-816e-ed24702ac107\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" Oct 30 23:27:36.947707 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:36.947650 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/709f007e-0ccc-40f5-816e-ed24702ac107-dshm\") pod \"distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj\" (UID: \"709f007e-0ccc-40f5-816e-ed24702ac107\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" Oct 30 23:27:36.947927 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:36.947821 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/709f007e-0ccc-40f5-816e-ed24702ac107-tls-certs\") pod \"distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj\" (UID: \"709f007e-0ccc-40f5-816e-ed24702ac107\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" Oct 30 23:27:37.002894 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:37.002817 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxvhm\" (UniqueName: \"kubernetes.io/projected/709f007e-0ccc-40f5-816e-ed24702ac107-kube-api-access-zxvhm\") pod \"distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj\" (UID: \"709f007e-0ccc-40f5-816e-ed24702ac107\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" Oct 30 23:27:37.030602 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:37.030568 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" Oct 30 23:27:37.117354 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:37.116547 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:27:37.503781 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:27:37.503749 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod709f007e_0ccc_40f5_816e_ed24702ac107.slice/crio-fdc042d8270ac99e7b7f7d5d17ffbc07e4f10512c8f90dd11239d31b5ecb911b WatchSource:0}: Error finding container fdc042d8270ac99e7b7f7d5d17ffbc07e4f10512c8f90dd11239d31b5ecb911b: Status 404 returned error can't find the container with id fdc042d8270ac99e7b7f7d5d17ffbc07e4f10512c8f90dd11239d31b5ecb911b Oct 30 23:27:37.504217 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:37.504147 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj"] Oct 30 23:27:38.038636 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:38.038601 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" event={"ID":"709f007e-0ccc-40f5-816e-ed24702ac107","Type":"ContainerStarted","Data":"1e48a65de5154a3b6c7e40cb316d8053b648d517b2451d436ab8cee282a5ad6f"} Oct 30 23:27:38.039131 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:38.038649 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" event={"ID":"709f007e-0ccc-40f5-816e-ed24702ac107","Type":"ContainerStarted","Data":"fdc042d8270ac99e7b7f7d5d17ffbc07e4f10512c8f90dd11239d31b5ecb911b"} Oct 30 23:27:38.153774 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:38.153728 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" podStartSLOduration=2.153714319 podStartE2EDuration="2.153714319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:27:38.152294831 +0000 UTC m=+1277.296125036" watchObservedRunningTime="2026-10-30 23:27:38.153714319 +0000 UTC m=+1277.297544523" Oct 30 23:27:39.030687 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:39.030660 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" Oct 30 23:27:39.675166 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:39.675112 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:27:40.478923 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:40.478889 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:27:41.708190 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:41.708121 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:27:43.269974 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.269940 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh"] Oct 30 23:27:43.274424 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.274404 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" Oct 30 23:27:43.371739 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.371709 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-istio-envoy\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" Oct 30 23:27:43.371931 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.371796 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-istio-podinfo\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" Oct 30 23:27:43.371931 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.371853 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-istiod-ca-cert\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" Oct 30 23:27:43.372092 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.372024 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-credential-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" Oct 30 23:27:43.372163 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.372110 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-istio-data\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" Oct 30 23:27:43.372163 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.372144 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-istio-token\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" Oct 30 23:27:43.372271 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.372227 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-workload-certs\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" Oct 30 23:27:43.372345 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.372319 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zrbk\" (UniqueName: \"kubernetes.io/projected/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-kube-api-access-5zrbk\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" Oct 30 23:27:43.372490 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.372425 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-workload-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" Oct 30 23:27:43.380892 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.380866 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh"] Oct 30 23:27:43.473567 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.473540 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-credential-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" Oct 30 23:27:43.473674 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.473574 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-istio-data\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" Oct 30 23:27:43.473674 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.473595 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-istio-token\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" Oct 30 23:27:43.473759 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.473716 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-workload-certs\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" Oct 30 23:27:43.473829 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.473809 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zrbk\" (UniqueName: \"kubernetes.io/projected/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-kube-api-access-5zrbk\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" Oct 30 23:27:43.473896 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.473882 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-workload-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" Oct 30 23:27:43.473959 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.473906 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-credential-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" Oct 30 23:27:43.474016 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.473965 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-istio-envoy\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" Oct 30 23:27:43.474103 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.474013 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-istio-podinfo\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" Oct 30 23:27:43.474103 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.474030 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-istiod-ca-cert\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" Oct 30 23:27:43.474103 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.474086 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-workload-certs\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" Oct 30 23:27:43.474247 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.474027 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-istio-data\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" Oct 30 23:27:43.474310 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.474290 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-workload-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" Oct 30 23:27:43.474646 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.474623 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-istiod-ca-cert\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" Oct 30 23:27:43.476446 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.476424 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-istio-envoy\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" Oct 30 23:27:43.476544 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.476522 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-istio-podinfo\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" Oct 30 23:27:43.531351 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.531293 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-istio-token\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" Oct 30 23:27:43.531857 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.531835 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zrbk\" (UniqueName: \"kubernetes.io/projected/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-kube-api-access-5zrbk\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" Oct 30 23:27:43.591086 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.591062 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" Oct 30 23:27:43.911031 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.911001 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7"] Oct 30 23:27:43.916219 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.916203 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" Oct 30 23:27:43.980340 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.980309 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l4j8\" (UniqueName: \"kubernetes.io/projected/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-kube-api-access-4l4j8\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" Oct 30 23:27:43.980500 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.980352 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-istio-envoy\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" Oct 30 23:27:43.980564 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.980518 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-workload-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" Oct 30 23:27:43.980638 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.980563 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-credential-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" Oct 30 23:27:43.980747 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.980726 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-istiod-ca-cert\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" Oct 30 23:27:43.980874 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.980856 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-istio-podinfo\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" Oct 30 23:27:43.980995 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.980982 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-istio-token\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" Oct 30 23:27:43.981105 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.981071 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-workload-certs\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" Oct 30 23:27:43.981219 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:43.981205 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-istio-data\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" Oct 30 23:27:44.007624 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:44.007597 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7"] Oct 30 23:27:44.073254 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:44.073227 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh"] Oct 30 23:27:44.073901 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:27:44.073872 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4257e26_e76e_49cc_bbd5_27c7e8075a5e.slice/crio-d78165a3e0abe000363846ba9f6f290e37dc8c880d78fedf2b8170d66cc6fb0f WatchSource:0}: Error finding container d78165a3e0abe000363846ba9f6f290e37dc8c880d78fedf2b8170d66cc6fb0f: Status 404 returned error can't find the container with id d78165a3e0abe000363846ba9f6f290e37dc8c880d78fedf2b8170d66cc6fb0f Oct 30 23:27:44.076382 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:44.076351 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:27:44.076472 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:44.076410 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:27:44.076472 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:44.076439 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:27:44.082022 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:44.082000 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-istiod-ca-cert\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" Oct 30 23:27:44.082140 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:44.082124 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-istio-podinfo\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" Oct 30 23:27:44.082205 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:44.082178 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-istio-token\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" Oct 30 23:27:44.082273 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:44.082256 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-workload-certs\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" Oct 30 23:27:44.082331 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:44.082295 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-istio-data\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" Oct 30 23:27:44.082387 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:44.082342 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l4j8\" (UniqueName: \"kubernetes.io/projected/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-kube-api-access-4l4j8\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" Oct 30 23:27:44.082441 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:44.082383 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-istio-envoy\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" Oct 30 23:27:44.082492 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:44.082460 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-workload-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" Oct 30 23:27:44.082492 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:44.082485 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-credential-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" Oct 30 23:27:44.082727 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:44.082704 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-workload-certs\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" Oct 30 23:27:44.082813 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:44.082789 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-istiod-ca-cert\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" Oct 30 23:27:44.082876 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:44.082840 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-credential-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" Oct 30 23:27:44.083050 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:44.082992 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-workload-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" Oct 30 23:27:44.083227 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:44.083199 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-istio-data\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" Oct 30 23:27:44.085336 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:44.085311 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-istio-envoy\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" Oct 30 23:27:44.085561 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:44.085542 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-istio-podinfo\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" Oct 30 23:27:44.104498 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:44.104474 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" event={"ID":"d4257e26-e76e-49cc-bbd5-27c7e8075a5e","Type":"ContainerStarted","Data":"d78165a3e0abe000363846ba9f6f290e37dc8c880d78fedf2b8170d66cc6fb0f"} Oct 30 23:27:44.139149 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:44.139127 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-istio-token\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" Oct 30 23:27:44.139877 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:44.139858 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l4j8\" (UniqueName: \"kubernetes.io/projected/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-kube-api-access-4l4j8\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" Oct 30 23:27:44.230358 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:44.230281 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" Oct 30 23:27:44.684116 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:44.683759 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7"] Oct 30 23:27:44.697726 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:44.695439 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:27:44.697726 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:44.695529 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:27:44.697726 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:44.695571 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:27:45.115355 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:45.115312 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" event={"ID":"3f222119-3b8c-4f8a-8ba2-88ef4d873da4","Type":"ContainerStarted","Data":"9a1e81f25be020d55e9bfe7661664bb4e1d5bbc2489d4fba85dc0bf6d6d82736"} Oct 30 23:27:45.115549 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:45.115382 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" event={"ID":"3f222119-3b8c-4f8a-8ba2-88ef4d873da4","Type":"ContainerStarted","Data":"225c22f8be1ec519a48926eac724d211661cb57c3fa1cbb99fbdbef90ef62738"} Oct 30 23:27:45.117022 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:45.116988 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" event={"ID":"d4257e26-e76e-49cc-bbd5-27c7e8075a5e","Type":"ContainerStarted","Data":"1420dfd58e348bcddbeaaa851e634d90f0afff6d55206efce39eefad251555b7"} Oct 30 23:27:45.230907 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:45.230878 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" Oct 30 23:27:45.232789 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:45.232764 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 23:27:45.232906 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:45.232813 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" podUID="3f222119-3b8c-4f8a-8ba2-88ef4d873da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 23:27:45.238963 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:45.238915 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" podStartSLOduration=2.238899032 podStartE2EDuration="2.238899032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:27:45.237791519 +0000 UTC m=+1284.381621729" watchObservedRunningTime="2026-10-30 23:27:45.238899032 +0000 UTC m=+1284.382729238" Oct 30 23:27:45.358221 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:45.358162 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" podStartSLOduration=2.358147636 podStartE2EDuration="2.358147636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:27:45.357282839 +0000 UTC m=+1284.501113063" watchObservedRunningTime="2026-10-30 23:27:45.358147636 +0000 UTC m=+1284.501977841" Oct 30 23:27:45.593422 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:45.593384 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" Oct 30 23:27:45.599691 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:45.599660 4952 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 23:27:45.599852 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:45.599725 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" podUID="d4257e26-e76e-49cc-bbd5-27c7e8075a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 23:27:46.233279 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:46.233242 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 23:27:46.233830 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:46.233297 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" podUID="3f222119-3b8c-4f8a-8ba2-88ef4d873da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 23:27:46.597504 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:46.597464 4952 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 23:27:46.597824 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:46.597784 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" podUID="d4257e26-e76e-49cc-bbd5-27c7e8075a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 23:27:46.699072 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:46.698995 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:27:47.008130 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:47.005737 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:27:47.056877 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:47.056843 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" Oct 30 23:27:47.143951 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:47.143024 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" Oct 30 23:27:47.159775 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:47.159748 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" Oct 30 23:27:47.231881 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:47.231449 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 23:27:47.231881 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:47.231510 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" podUID="3f222119-3b8c-4f8a-8ba2-88ef4d873da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 23:27:47.596072 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:47.591541 4952 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 23:27:47.596072 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:47.591618 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" podUID="d4257e26-e76e-49cc-bbd5-27c7e8075a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 23:27:48.231177 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:48.231143 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 23:27:48.231375 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:48.231220 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" podUID="3f222119-3b8c-4f8a-8ba2-88ef4d873da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 23:27:48.602320 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:48.602288 4952 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 23:27:48.602892 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:48.602860 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" podUID="d4257e26-e76e-49cc-bbd5-27c7e8075a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 23:27:49.231394 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:49.231361 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 23:27:49.231546 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:49.231412 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" podUID="3f222119-3b8c-4f8a-8ba2-88ef4d873da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 23:27:49.591428 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:49.591394 4952 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 23:27:49.591721 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:49.591692 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" podUID="d4257e26-e76e-49cc-bbd5-27c7e8075a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 23:27:50.231206 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:50.231169 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 23:27:50.231641 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:50.231235 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" podUID="3f222119-3b8c-4f8a-8ba2-88ef4d873da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 23:27:50.594461 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:50.594428 4952 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 23:27:50.594752 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:50.594723 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" podUID="d4257e26-e76e-49cc-bbd5-27c7e8075a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 23:27:51.144981 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:27:51.144943 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:27:51.231286 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:51.231248 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 23:27:51.231824 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:51.231316 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" podUID="3f222119-3b8c-4f8a-8ba2-88ef4d873da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 23:27:51.652499 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:51.652466 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" Oct 30 23:27:51.652845 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:51.652819 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" Oct 30 23:27:51.655572 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:51.655552 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" Oct 30 23:27:51.919141 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:51.918130 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:27:52.231376 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:52.231289 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 23:27:52.231754 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:52.231366 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" podUID="3f222119-3b8c-4f8a-8ba2-88ef4d873da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 23:27:53.235362 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:53.235321 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" Oct 30 23:27:53.235843 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:53.235700 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" Oct 30 23:27:53.236447 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:53.236427 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" Oct 30 23:27:56.684147 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:56.684104 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:27:59.600212 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:59.600181 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287"] Oct 30 23:27:59.605326 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:59.605306 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" Oct 30 23:27:59.662805 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:59.662776 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287"] Oct 30 23:27:59.707683 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:59.707655 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b81a7a-openshift-default-dockercfg-btsst\"" Oct 30 23:27:59.729813 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:59.729785 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-istio-data\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" Oct 30 23:27:59.729966 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:59.729836 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-workload-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" Oct 30 23:27:59.729966 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:59.729949 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-istio-podinfo\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" Oct 30 23:27:59.730129 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:59.730015 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-istio-token\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" Oct 30 23:27:59.730129 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:59.730115 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdfdl\" (UniqueName: \"kubernetes.io/projected/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-kube-api-access-qdfdl\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" Oct 30 23:27:59.730212 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:59.730168 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-istio-envoy\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" Oct 30 23:27:59.730212 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:59.730187 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-istiod-ca-cert\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" Oct 30 23:27:59.730314 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:59.730295 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-workload-certs\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" Oct 30 23:27:59.730361 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:59.730331 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-credential-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" Oct 30 23:27:59.831879 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:59.831852 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-workload-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" Oct 30 23:27:59.832425 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:59.831905 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-istio-podinfo\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" Oct 30 23:27:59.832425 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:59.831941 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-istio-token\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" Oct 30 23:27:59.832425 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:59.831984 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdfdl\" (UniqueName: \"kubernetes.io/projected/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-kube-api-access-qdfdl\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" Oct 30 23:27:59.832425 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:59.832063 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-istio-envoy\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" Oct 30 23:27:59.832425 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:59.832087 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-istiod-ca-cert\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" Oct 30 23:27:59.832425 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:59.832160 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-workload-certs\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" Oct 30 23:27:59.832425 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:59.832199 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-credential-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" Oct 30 23:27:59.832425 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:59.832269 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-istio-data\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" Oct 30 23:27:59.832425 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:59.832303 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-workload-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" Oct 30 23:27:59.832922 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:59.832757 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-credential-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" Oct 30 23:27:59.833165 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:59.832932 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-istiod-ca-cert\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" Oct 30 23:27:59.833165 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:59.833013 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-istio-data\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" Oct 30 23:27:59.833313 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:59.833214 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-workload-certs\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" Oct 30 23:27:59.834864 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:59.834837 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-istio-podinfo\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" Oct 30 23:27:59.835013 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:59.834997 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-istio-envoy\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" Oct 30 23:27:59.888147 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:59.888079 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdfdl\" (UniqueName: \"kubernetes.io/projected/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-kube-api-access-qdfdl\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" Oct 30 23:27:59.888254 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:59.888184 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-istio-token\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" Oct 30 23:27:59.919797 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:27:59.919751 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" Oct 30 23:28:00.355432 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:28:00.355400 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ecaaa2d_a6ac_4efc_ac36_559fa792d3b0.slice/crio-c9e4adb5e77e0688186812781eeec9d8098bf299f7ddaf2b897066cc3be80eaa WatchSource:0}: Error finding container c9e4adb5e77e0688186812781eeec9d8098bf299f7ddaf2b897066cc3be80eaa: Status 404 returned error can't find the container with id c9e4adb5e77e0688186812781eeec9d8098bf299f7ddaf2b897066cc3be80eaa Oct 30 23:28:00.356557 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:00.356500 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287"] Oct 30 23:28:00.358378 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:00.358343 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:28:00.358483 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:00.358426 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:28:00.358548 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:00.358477 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:28:01.298748 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:01.298706 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" event={"ID":"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0","Type":"ContainerStarted","Data":"9d2acd35e934df2a372acbb25ca4f11629e3ab8af05df898cfd5fdd4993ded71"} Oct 30 23:28:01.299189 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:01.298775 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" event={"ID":"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0","Type":"ContainerStarted","Data":"c9e4adb5e77e0688186812781eeec9d8098bf299f7ddaf2b897066cc3be80eaa"} Oct 30 23:28:01.449896 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:01.449847 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" podStartSLOduration=2.449832608 podStartE2EDuration="2.449832608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:28:01.446692125 +0000 UTC m=+1300.590522348" watchObservedRunningTime="2026-10-30 23:28:01.449832608 +0000 UTC m=+1300.593662812" Oct 30 23:28:01.488758 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:01.488608 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:28:01.920370 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:01.920331 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" Oct 30 23:28:01.921796 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:01.921763 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 23:28:01.921912 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:01.921808 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" podUID="2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 23:28:02.920680 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:02.920645 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 23:28:02.921081 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:02.920703 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" podUID="2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 23:28:03.920508 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:03.920472 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 23:28:03.920688 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:03.920522 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" podUID="2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 23:28:04.920221 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:04.920189 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 23:28:04.920610 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:04.920240 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" podUID="2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 23:28:05.921083 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:05.921018 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 23:28:05.921496 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:05.921101 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" podUID="2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 23:28:06.405736 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:06.405695 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:28:06.920757 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:06.920722 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 23:28:06.920964 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:06.920791 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" podUID="2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 23:28:07.924343 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:07.924315 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" Oct 30 23:28:07.924791 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:07.924774 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" Oct 30 23:28:07.925481 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:07.925459 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" Oct 30 23:28:11.063351 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:11.063322 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:28:11.285682 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:11.285643 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:28:15.167063 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:15.164847 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" podUID="da4935dd-c148-4cc4-84ab-1bfd91cce9b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 23:28:15.564222 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:15.564119 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" podUID="44adfd6a-2626-4378-bca1-d46ee1670b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:28:15.697208 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:15.697163 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" podUID="86ec338e-ad13-4029-a1cc-88650c6a27ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 23:28:16.589944 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:16.589907 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:28:17.948007 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:17.947976 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:28:21.145704 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:28:21.145674 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:28:21.387140 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:21.387110 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:28:26.184461 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:26.184410 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:28:31.080099 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:31.080054 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:28:34.759465 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:34.759430 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:28:35.898420 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:35.898378 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:28:37.140069 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:37.128315 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" podUID="da4935dd-c148-4cc4-84ab-1bfd91cce9b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:28:37.150204 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:37.150164 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" podUID="ab5286a5-6020-4c0b-9511-db63d4472a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:28:37.252391 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:37.223161 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" podUID="709f007e-0ccc-40f5-816e-ed24702ac107" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 23:28:40.873388 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:40.873351 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:28:43.439317 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:43.439273 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4"] Oct 30 23:28:43.444503 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:43.444479 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" Oct 30 23:28:43.533933 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:43.533904 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-istio-data\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" Oct 30 23:28:43.534130 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:43.533981 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-istio-envoy\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" Oct 30 23:28:43.534130 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:43.534026 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-workload-certs\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" Oct 30 23:28:43.534250 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:43.534152 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-istio-podinfo\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" Oct 30 23:28:43.534250 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:43.534189 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-istiod-ca-cert\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" Oct 30 23:28:43.534250 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:43.534228 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-workload-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" Oct 30 23:28:43.534867 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:43.534838 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-istio-token\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" Oct 30 23:28:43.535114 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:43.535086 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kgzh\" (UniqueName: \"kubernetes.io/projected/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-kube-api-access-9kgzh\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" Oct 30 23:28:43.535235 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:43.535152 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-credential-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" Oct 30 23:28:43.550663 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:43.550636 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4"] Oct 30 23:28:43.636692 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:43.636659 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-istiod-ca-cert\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" Oct 30 23:28:43.636835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:43.636720 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-workload-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" Oct 30 23:28:43.636835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:43.636779 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-istio-token\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" Oct 30 23:28:43.636835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:43.636794 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kgzh\" (UniqueName: \"kubernetes.io/projected/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-kube-api-access-9kgzh\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" Oct 30 23:28:43.636835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:43.636823 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-credential-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" Oct 30 23:28:43.637031 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:43.636890 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-istio-data\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" Oct 30 23:28:43.637031 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:43.636952 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-istio-envoy\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" Oct 30 23:28:43.637031 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:43.636990 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-workload-certs\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" Oct 30 23:28:43.637243 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:43.637059 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-istio-podinfo\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" Oct 30 23:28:43.637298 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:43.637250 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-workload-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" Oct 30 23:28:43.637384 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:43.637350 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-credential-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" Oct 30 23:28:43.637443 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:43.637423 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-workload-certs\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" Oct 30 23:28:43.637495 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:43.637478 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-istio-data\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" Oct 30 23:28:43.637534 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:43.637512 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-istiod-ca-cert\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" Oct 30 23:28:43.639547 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:43.639523 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-istio-envoy\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" Oct 30 23:28:43.639714 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:43.639697 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-istio-podinfo\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" Oct 30 23:28:43.693823 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:43.693756 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-istio-token\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" Oct 30 23:28:43.693934 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:43.693835 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kgzh\" (UniqueName: \"kubernetes.io/projected/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-kube-api-access-9kgzh\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" Oct 30 23:28:43.758593 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:43.758561 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" Oct 30 23:28:44.203721 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.203690 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4"] Oct 30 23:28:44.207449 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:28:44.207421 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd75821f5_34cd_4eeb_a9cc_a0c19573f0f8.slice/crio-a7a9e71b34154b4f92af06d6d09dc9152a6cb0b3b3559e747791427932a1aaa1 WatchSource:0}: Error finding container a7a9e71b34154b4f92af06d6d09dc9152a6cb0b3b3559e747791427932a1aaa1: Status 404 returned error can't find the container with id a7a9e71b34154b4f92af06d6d09dc9152a6cb0b3b3559e747791427932a1aaa1 Oct 30 23:28:44.209927 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.209887 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:28:44.210019 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.209982 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:28:44.210167 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.210030 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:28:44.604967 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.604935 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15b85996-b299h"] Oct 30 23:28:44.609760 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.609739 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b85996-b299h" Oct 30 23:28:44.654668 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.654612 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35a5425f-7388-4b75-a39e-827ae73f8be6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15b85996-b299h\" (UID: \"35a5425f-7388-4b75-a39e-827ae73f8be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b85996-b299h" Oct 30 23:28:44.654826 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.654733 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35a5425f-7388-4b75-a39e-827ae73f8be6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15b85996-b299h\" (UID: \"35a5425f-7388-4b75-a39e-827ae73f8be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b85996-b299h" Oct 30 23:28:44.654902 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.654828 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47mbm\" (UniqueName: \"kubernetes.io/projected/35a5425f-7388-4b75-a39e-827ae73f8be6-kube-api-access-47mbm\") pod \"maas-api-revoke-sub-15b85996-b299h\" (UID: \"35a5425f-7388-4b75-a39e-827ae73f8be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b85996-b299h" Oct 30 23:28:44.662126 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.662096 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:28:44.662396 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.662377 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\"" Oct 30 23:28:44.663366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.663335 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15b85996-b299h"] Oct 30 23:28:44.714762 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.714732 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj"] Oct 30 23:28:44.719813 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.719791 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" Oct 30 23:28:44.756870 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.756839 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35a5425f-7388-4b75-a39e-827ae73f8be6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15b85996-b299h\" (UID: \"35a5425f-7388-4b75-a39e-827ae73f8be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b85996-b299h" Oct 30 23:28:44.757025 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.756967 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35a5425f-7388-4b75-a39e-827ae73f8be6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15b85996-b299h\" (UID: \"35a5425f-7388-4b75-a39e-827ae73f8be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b85996-b299h" Oct 30 23:28:44.757025 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.756993 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47mbm\" (UniqueName: \"kubernetes.io/projected/35a5425f-7388-4b75-a39e-827ae73f8be6-kube-api-access-47mbm\") pod \"maas-api-revoke-sub-15b85996-b299h\" (UID: \"35a5425f-7388-4b75-a39e-827ae73f8be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b85996-b299h" Oct 30 23:28:44.757743 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.757716 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35a5425f-7388-4b75-a39e-827ae73f8be6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15b85996-b299h\" (UID: \"35a5425f-7388-4b75-a39e-827ae73f8be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b85996-b299h" Oct 30 23:28:44.771838 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.771815 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj"] Oct 30 23:28:44.773471 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.773448 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35a5425f-7388-4b75-a39e-827ae73f8be6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15b85996-b299h\" (UID: \"35a5425f-7388-4b75-a39e-827ae73f8be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b85996-b299h" Oct 30 23:28:44.791001 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.790974 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" event={"ID":"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8","Type":"ContainerStarted","Data":"cff7a5c11e1ce818e4db267beceec569eb4bbe27eac34c76ce518b1b3eee9ae3"} Oct 30 23:28:44.791209 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.791188 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" event={"ID":"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8","Type":"ContainerStarted","Data":"a7a9e71b34154b4f92af06d6d09dc9152a6cb0b3b3559e747791427932a1aaa1"} Oct 30 23:28:44.814337 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.814307 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47mbm\" (UniqueName: \"kubernetes.io/projected/35a5425f-7388-4b75-a39e-827ae73f8be6-kube-api-access-47mbm\") pod \"maas-api-revoke-sub-15b85996-b299h\" (UID: \"35a5425f-7388-4b75-a39e-827ae73f8be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b85996-b299h" Oct 30 23:28:44.857889 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.857832 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/482b6308-b55b-445f-b1a4-f95bab24f844-istio-data\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" Oct 30 23:28:44.857994 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.857944 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/482b6308-b55b-445f-b1a4-f95bab24f844-istio-envoy\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" Oct 30 23:28:44.857994 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.857968 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/482b6308-b55b-445f-b1a4-f95bab24f844-istiod-ca-cert\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" Oct 30 23:28:44.858110 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.858008 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/482b6308-b55b-445f-b1a4-f95bab24f844-workload-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" Oct 30 23:28:44.858192 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.858173 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/482b6308-b55b-445f-b1a4-f95bab24f844-workload-certs\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" Oct 30 23:28:44.858275 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.858259 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/482b6308-b55b-445f-b1a4-f95bab24f844-istio-token\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" Oct 30 23:28:44.858360 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.858341 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rprf\" (UniqueName: \"kubernetes.io/projected/482b6308-b55b-445f-b1a4-f95bab24f844-kube-api-access-8rprf\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" Oct 30 23:28:44.858456 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.858443 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/482b6308-b55b-445f-b1a4-f95bab24f844-istio-podinfo\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" Oct 30 23:28:44.858571 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.858554 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/482b6308-b55b-445f-b1a4-f95bab24f844-credential-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" Oct 30 23:28:44.924309 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.924276 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b85996-b299h" Oct 30 23:28:44.965540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.964960 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rprf\" (UniqueName: \"kubernetes.io/projected/482b6308-b55b-445f-b1a4-f95bab24f844-kube-api-access-8rprf\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" Oct 30 23:28:44.965540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.965060 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/482b6308-b55b-445f-b1a4-f95bab24f844-istio-podinfo\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" Oct 30 23:28:44.965540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.965181 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/482b6308-b55b-445f-b1a4-f95bab24f844-credential-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" Oct 30 23:28:44.965540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.965235 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/482b6308-b55b-445f-b1a4-f95bab24f844-istio-data\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" Oct 30 23:28:44.965540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.965335 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/482b6308-b55b-445f-b1a4-f95bab24f844-istio-envoy\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" Oct 30 23:28:44.965540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.965360 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/482b6308-b55b-445f-b1a4-f95bab24f844-istiod-ca-cert\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" Oct 30 23:28:44.965540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.965406 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/482b6308-b55b-445f-b1a4-f95bab24f844-workload-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" Oct 30 23:28:44.965540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.965455 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/482b6308-b55b-445f-b1a4-f95bab24f844-workload-certs\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" Oct 30 23:28:44.965540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.965507 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/482b6308-b55b-445f-b1a4-f95bab24f844-istio-token\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" Oct 30 23:28:44.979592 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.979506 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/482b6308-b55b-445f-b1a4-f95bab24f844-credential-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" Oct 30 23:28:44.979592 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.979514 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/482b6308-b55b-445f-b1a4-f95bab24f844-workload-certs\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" Oct 30 23:28:44.979839 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.979734 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/482b6308-b55b-445f-b1a4-f95bab24f844-istio-data\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" Oct 30 23:28:44.980150 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.980129 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/482b6308-b55b-445f-b1a4-f95bab24f844-istiod-ca-cert\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" Oct 30 23:28:44.983886 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:44.983777 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/482b6308-b55b-445f-b1a4-f95bab24f844-workload-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" Oct 30 23:28:45.005673 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:45.005615 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/482b6308-b55b-445f-b1a4-f95bab24f844-istio-envoy\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" Oct 30 23:28:45.006223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:45.006184 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/482b6308-b55b-445f-b1a4-f95bab24f844-istio-podinfo\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" Oct 30 23:28:45.022774 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:45.022749 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rprf\" (UniqueName: \"kubernetes.io/projected/482b6308-b55b-445f-b1a4-f95bab24f844-kube-api-access-8rprf\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" Oct 30 23:28:45.023252 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:45.023230 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/482b6308-b55b-445f-b1a4-f95bab24f844-istio-token\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" Oct 30 23:28:45.038752 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:45.038214 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" Oct 30 23:28:45.355361 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:45.355316 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" podStartSLOduration=2.355300048 podStartE2EDuration="2.355300048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:28:44.97682531 +0000 UTC m=+1344.120655542" watchObservedRunningTime="2026-10-30 23:28:45.355300048 +0000 UTC m=+1344.499130254" Oct 30 23:28:45.357947 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:28:45.357918 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a5425f_7388_4b75_a39e_827ae73f8be6.slice/crio-5fa4a7f7572e5468565d218ac98add2dcf63c77be90dcaef4c006a6045bbbfb9 WatchSource:0}: Error finding container 5fa4a7f7572e5468565d218ac98add2dcf63c77be90dcaef4c006a6045bbbfb9: Status 404 returned error can't find the container with id 5fa4a7f7572e5468565d218ac98add2dcf63c77be90dcaef4c006a6045bbbfb9 Oct 30 23:28:45.358154 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:45.358063 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15b85996-b299h"] Oct 30 23:28:45.494135 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:45.494104 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj"] Oct 30 23:28:45.495797 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:28:45.495773 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod482b6308_b55b_445f_b1a4_f95bab24f844.slice/crio-36d4e8b797b22674f75705979d2164e07ecc870b30bfc5a0a287a2c86c071f2f WatchSource:0}: Error finding container 36d4e8b797b22674f75705979d2164e07ecc870b30bfc5a0a287a2c86c071f2f: Status 404 returned error can't find the container with id 36d4e8b797b22674f75705979d2164e07ecc870b30bfc5a0a287a2c86c071f2f Oct 30 23:28:45.498450 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:45.498418 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:28:45.498514 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:45.498479 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:28:45.498514 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:45.498506 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:28:45.696844 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:45.696700 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:28:45.759647 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:45.759616 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" Oct 30 23:28:45.760640 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:45.760619 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 23:28:45.760812 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:45.760792 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" podUID="d75821f5-34cd-4eeb-a9cc-a0c19573f0f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 23:28:45.813355 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:45.812920 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" event={"ID":"482b6308-b55b-445f-b1a4-f95bab24f844","Type":"ContainerStarted","Data":"f09fd34c21cdd17cb1aa3ad23dc79b4840838bc9b358c9566c7fd598ffdabeee"} Oct 30 23:28:45.813548 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:45.813530 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" event={"ID":"482b6308-b55b-445f-b1a4-f95bab24f844","Type":"ContainerStarted","Data":"36d4e8b797b22674f75705979d2164e07ecc870b30bfc5a0a287a2c86c071f2f"} Oct 30 23:28:45.814976 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:45.814952 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b85996-b299h" event={"ID":"35a5425f-7388-4b75-a39e-827ae73f8be6","Type":"ContainerStarted","Data":"5fa4a7f7572e5468565d218ac98add2dcf63c77be90dcaef4c006a6045bbbfb9"} Oct 30 23:28:45.936586 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:45.936527 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" podStartSLOduration=1.936508581 podStartE2EDuration="1.936508581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:28:45.932921823 +0000 UTC m=+1345.076752053" watchObservedRunningTime="2026-10-30 23:28:45.936508581 +0000 UTC m=+1345.080338788" Oct 30 23:28:46.038927 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:46.038852 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" Oct 30 23:28:46.040181 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:46.040161 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 23:28:46.040298 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:46.040202 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" podUID="482b6308-b55b-445f-b1a4-f95bab24f844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 23:28:46.758854 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:46.758821 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 23:28:46.759347 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:46.758878 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" podUID="d75821f5-34cd-4eeb-a9cc-a0c19573f0f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 23:28:46.846813 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:46.846777 4952 generic.go:356] "Generic (PLEG): container finished" podID="35a5425f-7388-4b75-a39e-827ae73f8be6" containerID="f51bd76e1e584bfd6e696cb683cbc83b6c9afe982a6d0702b83d212e364fcc6f" exitCode=0 Oct 30 23:28:46.847503 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:46.847445 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b85996-b299h" event={"ID":"35a5425f-7388-4b75-a39e-827ae73f8be6","Type":"ContainerDied","Data":"f51bd76e1e584bfd6e696cb683cbc83b6c9afe982a6d0702b83d212e364fcc6f"} Oct 30 23:28:47.041886 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:47.040089 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 23:28:47.041886 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:47.040176 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" podUID="482b6308-b55b-445f-b1a4-f95bab24f844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 23:28:47.759843 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:47.759811 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 23:28:47.760356 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:47.759876 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" podUID="d75821f5-34cd-4eeb-a9cc-a0c19573f0f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 23:28:48.039331 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:48.038831 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 23:28:48.039331 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:48.038893 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" podUID="482b6308-b55b-445f-b1a4-f95bab24f844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 23:28:48.206274 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:48.206240 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b85996-b299h" Oct 30 23:28:48.254477 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:48.254447 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35a5425f-7388-4b75-a39e-827ae73f8be6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35a5425f-7388-4b75-a39e-827ae73f8be6-maas-api-xks-ca\") pod \"35a5425f-7388-4b75-a39e-827ae73f8be6\" (UID: \"35a5425f-7388-4b75-a39e-827ae73f8be6\") " Oct 30 23:28:48.254638 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:48.254504 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35a5425f-7388-4b75-a39e-827ae73f8be6-kube-api-access-47mbm\" (UniqueName: \"kubernetes.io/projected/35a5425f-7388-4b75-a39e-827ae73f8be6-kube-api-access-47mbm\") pod \"35a5425f-7388-4b75-a39e-827ae73f8be6\" (UID: \"35a5425f-7388-4b75-a39e-827ae73f8be6\") " Oct 30 23:28:48.254638 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:48.254549 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35a5425f-7388-4b75-a39e-827ae73f8be6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35a5425f-7388-4b75-a39e-827ae73f8be6-maas-api-openshift-ca\") pod \"35a5425f-7388-4b75-a39e-827ae73f8be6\" (UID: \"35a5425f-7388-4b75-a39e-827ae73f8be6\") " Oct 30 23:28:48.255664 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:48.255589 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35a5425f-7388-4b75-a39e-827ae73f8be6-maas-api-openshift-ca" pod "35a5425f-7388-4b75-a39e-827ae73f8be6" (UID: "35a5425f-7388-4b75-a39e-827ae73f8be6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:28:48.256898 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:48.256873 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a5425f-7388-4b75-a39e-827ae73f8be6-maas-api-xks-ca" pod "35a5425f-7388-4b75-a39e-827ae73f8be6" (UID: "35a5425f-7388-4b75-a39e-827ae73f8be6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:28:48.258169 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:48.258149 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a5425f-7388-4b75-a39e-827ae73f8be6-kube-api-access-47mbm" pod "35a5425f-7388-4b75-a39e-827ae73f8be6" (UID: "35a5425f-7388-4b75-a39e-827ae73f8be6"). InnerVolumeSpecName "kube-api-access-47mbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:28:48.357258 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:48.357223 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35a5425f-7388-4b75-a39e-827ae73f8be6-maas-api-xks-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:28:48.357258 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:48.357257 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47mbm\" (UniqueName: \"kubernetes.io/projected/35a5425f-7388-4b75-a39e-827ae73f8be6-kube-api-access-47mbm\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:28:48.357503 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:48.357273 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35a5425f-7388-4b75-a39e-827ae73f8be6-maas-api-openshift-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:28:48.759597 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:48.759522 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 23:28:48.759597 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:48.759579 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" podUID="d75821f5-34cd-4eeb-a9cc-a0c19573f0f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 23:28:48.864737 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:48.864708 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b85996-b299h" Oct 30 23:28:48.865160 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:48.864712 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b85996-b299h" event={"ID":"35a5425f-7388-4b75-a39e-827ae73f8be6","Type":"ContainerDied","Data":"5fa4a7f7572e5468565d218ac98add2dcf63c77be90dcaef4c006a6045bbbfb9"} Oct 30 23:28:48.865160 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:48.864829 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fa4a7f7572e5468565d218ac98add2dcf63c77be90dcaef4c006a6045bbbfb9" Oct 30 23:28:49.039462 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:49.039385 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 23:28:49.039607 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:49.039453 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" podUID="482b6308-b55b-445f-b1a4-f95bab24f844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 23:28:49.759683 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:49.759650 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 23:28:49.759837 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:49.759699 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" podUID="d75821f5-34cd-4eeb-a9cc-a0c19573f0f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 23:28:50.038899 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:50.038827 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 23:28:50.038899 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:50.038874 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" podUID="482b6308-b55b-445f-b1a4-f95bab24f844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 23:28:50.759781 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:50.759738 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 23:28:50.759946 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:50.759811 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" podUID="d75821f5-34cd-4eeb-a9cc-a0c19573f0f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 23:28:50.999564 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:50.999527 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:28:51.038954 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:51.038866 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 23:28:51.038954 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:51.038918 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" podUID="482b6308-b55b-445f-b1a4-f95bab24f844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 23:28:51.145285 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:28:51.145249 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:28:52.039720 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:52.039675 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 23:28:52.040238 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:52.039734 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" podUID="482b6308-b55b-445f-b1a4-f95bab24f844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 23:28:52.073542 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:52.073514 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" Oct 30 23:28:52.074784 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:52.074757 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" Oct 30 23:28:52.075543 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:52.075520 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" Oct 30 23:28:53.335136 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:53.335093 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" Oct 30 23:28:53.335919 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:53.335896 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" Oct 30 23:28:53.336578 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:53.336561 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" Oct 30 23:28:54.954252 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:54.954219 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:28:55.712868 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:55.712829 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:28:59.505362 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:59.505329 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t"] Oct 30 23:28:59.506151 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:59.506130 4952 cpu_manager.go:414] "containerMap: removing container" podUID="35a5425f-7388-4b75-a39e-827ae73f8be6" containerName="revoke-keys" Oct 30 23:28:59.506151 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:59.506151 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35a5425f-7388-4b75-a39e-827ae73f8be6" containerName="revoke-keys" Oct 30 23:28:59.506355 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:59.506341 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="35a5425f-7388-4b75-a39e-827ae73f8be6" containerName="revoke-keys" Oct 30 23:28:59.510400 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:59.510372 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" Oct 30 23:28:59.652120 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:59.652084 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-credential-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" Oct 30 23:28:59.652266 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:59.652134 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-istio-envoy\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" Oct 30 23:28:59.652266 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:59.652230 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd5kn\" (UniqueName: \"kubernetes.io/projected/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-kube-api-access-gd5kn\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" Oct 30 23:28:59.652339 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:59.652264 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-workload-certs\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" Oct 30 23:28:59.652402 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:59.652382 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-istio-podinfo\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" Oct 30 23:28:59.652455 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:59.652444 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-istiod-ca-cert\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" Oct 30 23:28:59.652490 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:59.652466 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-istio-token\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" Oct 30 23:28:59.652526 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:59.652487 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-workload-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" Oct 30 23:28:59.652589 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:59.652576 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-istio-data\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" Oct 30 23:28:59.709377 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:59.709349 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t"] Oct 30 23:28:59.754270 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:59.754240 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-istio-podinfo\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" Oct 30 23:28:59.754429 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:59.754338 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-istiod-ca-cert\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" Oct 30 23:28:59.754429 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:59.754369 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-istio-token\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" Oct 30 23:28:59.754553 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:59.754436 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-workload-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" Oct 30 23:28:59.754553 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:59.754521 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-istio-data\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" Oct 30 23:28:59.754661 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:59.754561 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-credential-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" Oct 30 23:28:59.754661 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:59.754616 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-istio-envoy\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" Oct 30 23:28:59.754788 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:59.754676 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd5kn\" (UniqueName: \"kubernetes.io/projected/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-kube-api-access-gd5kn\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" Oct 30 23:28:59.754788 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:59.754715 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-workload-certs\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" Oct 30 23:28:59.755093 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:59.755068 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-istio-data\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" Oct 30 23:28:59.755190 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:59.755161 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-workload-certs\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" Oct 30 23:28:59.755357 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:59.755334 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-workload-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" Oct 30 23:28:59.755636 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:59.755578 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-credential-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" Oct 30 23:28:59.755940 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:59.755920 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-istiod-ca-cert\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" Oct 30 23:28:59.764380 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:59.764359 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-istio-envoy\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" Oct 30 23:28:59.764644 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:59.764622 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-istio-podinfo\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" Oct 30 23:28:59.814460 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:59.814434 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd5kn\" (UniqueName: \"kubernetes.io/projected/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-kube-api-access-gd5kn\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" Oct 30 23:28:59.814584 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:59.814494 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-istio-token\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" Oct 30 23:28:59.825538 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:28:59.825509 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" Oct 30 23:29:00.138827 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.138798 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889"] Oct 30 23:29:00.144150 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.144132 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" Oct 30 23:29:00.230593 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.230447 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889"] Oct 30 23:29:00.263053 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.262994 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x28dr\" (UniqueName: \"kubernetes.io/projected/3ad4e638-0d0d-423b-96db-f671e8095f08-kube-api-access-x28dr\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" Oct 30 23:29:00.263237 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.263093 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad4e638-0d0d-423b-96db-f671e8095f08-credential-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" Oct 30 23:29:00.263306 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.263289 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad4e638-0d0d-423b-96db-f671e8095f08-istio-data\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" Oct 30 23:29:00.263363 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.263338 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad4e638-0d0d-423b-96db-f671e8095f08-workload-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" Oct 30 23:29:00.263425 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.263389 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad4e638-0d0d-423b-96db-f671e8095f08-workload-certs\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" Oct 30 23:29:00.263518 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.263499 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad4e638-0d0d-423b-96db-f671e8095f08-istio-envoy\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" Oct 30 23:29:00.263623 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.263608 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ad4e638-0d0d-423b-96db-f671e8095f08-istio-token\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" Oct 30 23:29:00.263690 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.263675 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad4e638-0d0d-423b-96db-f671e8095f08-istio-podinfo\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" Oct 30 23:29:00.263748 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.263705 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad4e638-0d0d-423b-96db-f671e8095f08-istiod-ca-cert\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" Oct 30 23:29:00.302548 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:29:00.302508 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec9e7fff_1e25_4f16_ab95_f1a1d65448da.slice/crio-4c099a90ac841770291f2a75ee197c79a5f94c8fee29f91109ff64b7c7fa9107 WatchSource:0}: Error finding container 4c099a90ac841770291f2a75ee197c79a5f94c8fee29f91109ff64b7c7fa9107: Status 404 returned error can't find the container with id 4c099a90ac841770291f2a75ee197c79a5f94c8fee29f91109ff64b7c7fa9107 Oct 30 23:29:00.303023 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.303000 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t"] Oct 30 23:29:00.305191 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.305151 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:29:00.305270 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.305217 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:29:00.305270 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.305250 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:29:00.365290 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.365267 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad4e638-0d0d-423b-96db-f671e8095f08-workload-certs\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" Oct 30 23:29:00.365388 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.365363 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad4e638-0d0d-423b-96db-f671e8095f08-istio-envoy\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" Oct 30 23:29:00.365439 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.365425 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ad4e638-0d0d-423b-96db-f671e8095f08-istio-token\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" Oct 30 23:29:00.365575 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.365551 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad4e638-0d0d-423b-96db-f671e8095f08-istio-podinfo\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" Oct 30 23:29:00.365639 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.365597 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad4e638-0d0d-423b-96db-f671e8095f08-istiod-ca-cert\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" Oct 30 23:29:00.365698 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.365671 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x28dr\" (UniqueName: \"kubernetes.io/projected/3ad4e638-0d0d-423b-96db-f671e8095f08-kube-api-access-x28dr\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" Oct 30 23:29:00.365759 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.365695 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad4e638-0d0d-423b-96db-f671e8095f08-workload-certs\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" Oct 30 23:29:00.365759 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.365737 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad4e638-0d0d-423b-96db-f671e8095f08-credential-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" Oct 30 23:29:00.365878 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.365798 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad4e638-0d0d-423b-96db-f671e8095f08-istio-data\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" Oct 30 23:29:00.365878 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.365844 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad4e638-0d0d-423b-96db-f671e8095f08-workload-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" Oct 30 23:29:00.366304 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.366277 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad4e638-0d0d-423b-96db-f671e8095f08-credential-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" Oct 30 23:29:00.366525 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.366397 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad4e638-0d0d-423b-96db-f671e8095f08-istio-data\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" Oct 30 23:29:00.366525 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.366477 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad4e638-0d0d-423b-96db-f671e8095f08-workload-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" Oct 30 23:29:00.366648 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.366612 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad4e638-0d0d-423b-96db-f671e8095f08-istiod-ca-cert\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" Oct 30 23:29:00.367873 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.367856 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad4e638-0d0d-423b-96db-f671e8095f08-istio-envoy\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" Oct 30 23:29:00.368250 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.368233 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad4e638-0d0d-423b-96db-f671e8095f08-istio-podinfo\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" Oct 30 23:29:00.422904 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.422821 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x28dr\" (UniqueName: \"kubernetes.io/projected/3ad4e638-0d0d-423b-96db-f671e8095f08-kube-api-access-x28dr\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" Oct 30 23:29:00.423029 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.422970 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ad4e638-0d0d-423b-96db-f671e8095f08-istio-token\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" Oct 30 23:29:00.459758 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.459735 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" Oct 30 23:29:00.655349 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.655312 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:00.898448 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.898416 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889"] Oct 30 23:29:00.898884 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:29:00.898851 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ad4e638_0d0d_423b_96db_f671e8095f08.slice/crio-e7c66ef553e45ff8f65f6e9c4a03cda12250821f14e15f213ece30c5a230fd4d WatchSource:0}: Error finding container e7c66ef553e45ff8f65f6e9c4a03cda12250821f14e15f213ece30c5a230fd4d: Status 404 returned error can't find the container with id e7c66ef553e45ff8f65f6e9c4a03cda12250821f14e15f213ece30c5a230fd4d Oct 30 23:29:00.919946 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.919908 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:29:00.920095 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.919972 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:29:00.920095 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:00.920001 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:29:01.010395 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:01.010361 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" event={"ID":"ec9e7fff-1e25-4f16-ab95-f1a1d65448da","Type":"ContainerStarted","Data":"59385393f120d44716737047ed32fab568f953953c10981e08ab28b4dcc48fbe"} Oct 30 23:29:01.010557 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:01.010418 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" event={"ID":"ec9e7fff-1e25-4f16-ab95-f1a1d65448da","Type":"ContainerStarted","Data":"4c099a90ac841770291f2a75ee197c79a5f94c8fee29f91109ff64b7c7fa9107"} Oct 30 23:29:01.012383 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:01.012358 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" event={"ID":"3ad4e638-0d0d-423b-96db-f671e8095f08","Type":"ContainerStarted","Data":"e7c66ef553e45ff8f65f6e9c4a03cda12250821f14e15f213ece30c5a230fd4d"} Oct 30 23:29:01.132499 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:01.132444 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" podStartSLOduration=2.132430793 podStartE2EDuration="2.132430793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:29:01.130464475 +0000 UTC m=+1360.274294680" watchObservedRunningTime="2026-10-30 23:29:01.132430793 +0000 UTC m=+1360.276260998" Oct 30 23:29:01.826457 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:01.826427 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" Oct 30 23:29:01.827802 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:01.827770 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 23:29:01.827948 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:01.827832 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" podUID="ec9e7fff-1e25-4f16-ab95-f1a1d65448da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 23:29:02.038573 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:02.038539 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" event={"ID":"3ad4e638-0d0d-423b-96db-f671e8095f08","Type":"ContainerStarted","Data":"6006616f914193bef622bbf1bb6966adf82da6c3fa10853b936b8ba9104d69e7"} Oct 30 23:29:02.192065 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:02.188901 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" podStartSLOduration=2.188876874 podStartE2EDuration="2.188876874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:29:02.163272223 +0000 UTC m=+1361.307102452" watchObservedRunningTime="2026-10-30 23:29:02.188876874 +0000 UTC m=+1361.332707079" Oct 30 23:29:02.460533 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:02.460434 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" Oct 30 23:29:02.461735 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:02.461714 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 23:29:02.461828 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:02.461763 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" podUID="3ad4e638-0d0d-423b-96db-f671e8095f08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 23:29:02.827800 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:02.827716 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 23:29:02.827800 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:02.827774 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" podUID="ec9e7fff-1e25-4f16-ab95-f1a1d65448da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 23:29:03.063084 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:03.060955 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cc6b31f-npwbj"] Oct 30 23:29:03.063084 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:03.060993 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cc6b31f-npwbj"] Oct 30 23:29:03.460792 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:03.460759 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 23:29:03.460983 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:03.460826 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" podUID="3ad4e638-0d0d-423b-96db-f671e8095f08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 23:29:03.826266 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:03.826185 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 23:29:03.826499 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:03.826465 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" podUID="ec9e7fff-1e25-4f16-ab95-f1a1d65448da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 23:29:04.460556 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:04.460518 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 23:29:04.460940 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:04.460597 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" podUID="3ad4e638-0d0d-423b-96db-f671e8095f08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 23:29:04.826715 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:04.826621 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 23:29:04.826715 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:04.826688 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" podUID="ec9e7fff-1e25-4f16-ab95-f1a1d65448da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 23:29:05.021769 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:05.021726 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="becfdd7d-d2f1-4a08-aeca-275a4a0e40b2" path="/var/lib/kubelet/pods/becfdd7d-d2f1-4a08-aeca-275a4a0e40b2/volumes" Oct 30 23:29:05.460792 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:05.460757 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 23:29:05.461356 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:05.460823 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" podUID="3ad4e638-0d0d-423b-96db-f671e8095f08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 23:29:05.826563 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:05.826469 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 23:29:05.826846 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:05.826809 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" podUID="ec9e7fff-1e25-4f16-ab95-f1a1d65448da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 23:29:05.857245 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:05.857209 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:06.460778 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:06.460741 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 23:29:06.460988 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:06.460806 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" podUID="3ad4e638-0d0d-423b-96db-f671e8095f08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 23:29:06.826625 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:06.826544 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 23:29:06.826768 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:06.826621 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" podUID="ec9e7fff-1e25-4f16-ab95-f1a1d65448da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 23:29:07.460131 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:07.460094 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 23:29:07.460337 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:07.460170 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" podUID="3ad4e638-0d0d-423b-96db-f671e8095f08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 23:29:08.043735 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:08.043703 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" Oct 30 23:29:08.044298 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:08.044078 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" Oct 30 23:29:08.045353 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:08.045330 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" Oct 30 23:29:08.616927 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:08.616883 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" Oct 30 23:29:08.617408 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:08.617384 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" Oct 30 23:29:08.618373 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:08.618353 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" Oct 30 23:29:10.765651 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:10.765612 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:14.270150 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:14.268014 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" podUID="959b92af-9035-4dbd-aed3-5db4daaa63a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:14.570608 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:14.570518 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" podUID="44adfd6a-2626-4378-bca1-d46ee1670b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:14.779770 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:14.779727 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" podUID="6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:14.913953 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:14.913907 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" podUID="ab5286a5-6020-4c0b-9511-db63d4472a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:15.031060 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:15.028182 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" podUID="44adfd6a-2626-4378-bca1-d46ee1670b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:15.123198 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:15.120111 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" podUID="da4935dd-c148-4cc4-84ab-1bfd91cce9b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 23:29:15.169142 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:15.160137 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" podUID="709f007e-0ccc-40f5-816e-ed24702ac107" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:15.292066 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:15.272486 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" podUID="959b92af-9035-4dbd-aed3-5db4daaa63a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:15.292066 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:15.273420 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" Oct 30 23:29:15.497119 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:15.484543 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp"] Oct 30 23:29:15.784257 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:15.781145 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" podUID="6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:15.869841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:15.869813 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" Oct 30 23:29:15.870479 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:15.870373 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" Oct 30 23:29:15.980062 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:15.964176 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/553c8b63-e975-498f-9313-878fbf99066d-istio-podinfo\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" Oct 30 23:29:15.980062 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:15.964301 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spvwr\" (UniqueName: \"kubernetes.io/projected/553c8b63-e975-498f-9313-878fbf99066d-kube-api-access-spvwr\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" Oct 30 23:29:15.980062 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:15.964363 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/553c8b63-e975-498f-9313-878fbf99066d-istio-envoy\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" Oct 30 23:29:15.980062 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:15.964400 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/553c8b63-e975-498f-9313-878fbf99066d-workload-certs\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" Oct 30 23:29:15.980062 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:15.964447 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/553c8b63-e975-498f-9313-878fbf99066d-workload-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" Oct 30 23:29:15.980062 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:15.964560 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/553c8b63-e975-498f-9313-878fbf99066d-istio-token\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" Oct 30 23:29:15.980062 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:15.964585 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/553c8b63-e975-498f-9313-878fbf99066d-istiod-ca-cert\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" Oct 30 23:29:15.980062 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:15.964638 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/553c8b63-e975-498f-9313-878fbf99066d-credential-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" Oct 30 23:29:15.980062 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:15.964670 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/553c8b63-e975-498f-9313-878fbf99066d-istio-data\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" Oct 30 23:29:15.990356 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:15.990320 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" podUID="43e39353-ae85-4d76-9900-7e7927409098" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:16.025534 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.011222 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" podUID="ab5286a5-6020-4c0b-9511-db63d4472a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 23:29:16.025534 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.012530 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" Oct 30 23:29:16.027611 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.027528 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp"] Oct 30 23:29:16.164572 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.078208 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spvwr\" (UniqueName: \"kubernetes.io/projected/553c8b63-e975-498f-9313-878fbf99066d-kube-api-access-spvwr\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" Oct 30 23:29:16.164572 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.078291 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/553c8b63-e975-498f-9313-878fbf99066d-istio-envoy\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" Oct 30 23:29:16.164572 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.078334 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/553c8b63-e975-498f-9313-878fbf99066d-workload-certs\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" Oct 30 23:29:16.164572 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.078386 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/553c8b63-e975-498f-9313-878fbf99066d-workload-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" Oct 30 23:29:16.164572 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.078476 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/553c8b63-e975-498f-9313-878fbf99066d-istio-token\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" Oct 30 23:29:16.164572 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.078500 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/553c8b63-e975-498f-9313-878fbf99066d-istiod-ca-cert\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" Oct 30 23:29:16.164572 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.078551 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/553c8b63-e975-498f-9313-878fbf99066d-credential-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" Oct 30 23:29:16.164572 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.078585 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/553c8b63-e975-498f-9313-878fbf99066d-istio-data\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" Oct 30 23:29:16.164572 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.078690 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/553c8b63-e975-498f-9313-878fbf99066d-istio-podinfo\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" Oct 30 23:29:16.164572 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.085085 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/553c8b63-e975-498f-9313-878fbf99066d-workload-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" Oct 30 23:29:16.164572 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.087183 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/553c8b63-e975-498f-9313-878fbf99066d-istiod-ca-cert\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" Oct 30 23:29:16.164572 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.087715 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/553c8b63-e975-498f-9313-878fbf99066d-workload-certs\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" Oct 30 23:29:16.164572 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.087957 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/553c8b63-e975-498f-9313-878fbf99066d-credential-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" Oct 30 23:29:16.196653 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.191291 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" podUID="da4935dd-c148-4cc4-84ab-1bfd91cce9b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 23:29:16.196653 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.192891 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" Oct 30 23:29:16.329546 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.273552 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" podUID="959b92af-9035-4dbd-aed3-5db4daaa63a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:16.329546 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.302259 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/553c8b63-e975-498f-9313-878fbf99066d-istio-envoy\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" Oct 30 23:29:16.329546 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.304843 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" podUID="709f007e-0ccc-40f5-816e-ed24702ac107" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:16.329546 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.320810 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" podUID="86a57a21-72bc-4ce4-8d1e-bf29214ad4de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 30 23:29:16.329546 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.322626 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" Oct 30 23:29:16.329546 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.324186 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spvwr\" (UniqueName: \"kubernetes.io/projected/553c8b63-e975-498f-9313-878fbf99066d-kube-api-access-spvwr\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" Oct 30 23:29:16.329546 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.326433 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/553c8b63-e975-498f-9313-878fbf99066d-istio-data\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" Oct 30 23:29:16.523836 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.523754 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:16.647061 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.634994 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" podUID="44adfd6a-2626-4378-bca1-d46ee1670b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:16.672063 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.637897 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" Oct 30 23:29:16.682625 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.660727 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/553c8b63-e975-498f-9313-878fbf99066d-istio-podinfo\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" Oct 30 23:29:16.694614 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.694584 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" Oct 30 23:29:16.741178 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.738567 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/553c8b63-e975-498f-9313-878fbf99066d-istio-token\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" Oct 30 23:29:16.743059 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.741456 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" Oct 30 23:29:16.793106 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.791120 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" podUID="28913940-90f7-4d80-89b7-f5eece975d02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:16.797027 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.795833 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" Oct 30 23:29:16.848083 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.847160 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" podUID="145e7c01-efec-46e6-a748-56b39de76f83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 23:29:16.862436 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.862405 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf56d53c-wphkc"] Oct 30 23:29:16.882540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.868085 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf56d53c-wphkc" Oct 30 23:29:16.949975 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.923544 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:29:16.949975 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:16.923701 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\"" Oct 30 23:29:17.020569 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:17.015128 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" podUID="ab5286a5-6020-4c0b-9511-db63d4472a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:17.025294 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:17.024900 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e89ad741-5b63-4713-8c79-698219b308e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf56d53c-wphkc\" (UID: \"e89ad741-5b63-4713-8c79-698219b308e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf56d53c-wphkc" Oct 30 23:29:17.025294 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:17.025009 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e89ad741-5b63-4713-8c79-698219b308e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf56d53c-wphkc\" (UID: \"e89ad741-5b63-4713-8c79-698219b308e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf56d53c-wphkc" Oct 30 23:29:17.025783 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:17.025602 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwtst\" (UniqueName: \"kubernetes.io/projected/e89ad741-5b63-4713-8c79-698219b308e0-kube-api-access-rwtst\") pod \"maas-api-revoke-sub-bf56d53c-wphkc\" (UID: \"e89ad741-5b63-4713-8c79-698219b308e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf56d53c-wphkc" Oct 30 23:29:17.128565 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:17.126669 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e89ad741-5b63-4713-8c79-698219b308e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf56d53c-wphkc\" (UID: \"e89ad741-5b63-4713-8c79-698219b308e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf56d53c-wphkc" Oct 30 23:29:17.128565 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:17.126806 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwtst\" (UniqueName: \"kubernetes.io/projected/e89ad741-5b63-4713-8c79-698219b308e0-kube-api-access-rwtst\") pod \"maas-api-revoke-sub-bf56d53c-wphkc\" (UID: \"e89ad741-5b63-4713-8c79-698219b308e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf56d53c-wphkc" Oct 30 23:29:17.128565 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:17.127080 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e89ad741-5b63-4713-8c79-698219b308e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf56d53c-wphkc\" (UID: \"e89ad741-5b63-4713-8c79-698219b308e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf56d53c-wphkc" Oct 30 23:29:17.128565 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:17.128127 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e89ad741-5b63-4713-8c79-698219b308e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf56d53c-wphkc\" (UID: \"e89ad741-5b63-4713-8c79-698219b308e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf56d53c-wphkc" Oct 30 23:29:17.224507 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:17.224475 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwtst\" (UniqueName: \"kubernetes.io/projected/e89ad741-5b63-4713-8c79-698219b308e0-kube-api-access-rwtst\") pod \"maas-api-revoke-sub-bf56d53c-wphkc\" (UID: \"e89ad741-5b63-4713-8c79-698219b308e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf56d53c-wphkc" Oct 30 23:29:17.316144 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:17.315973 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e89ad741-5b63-4713-8c79-698219b308e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf56d53c-wphkc\" (UID: \"e89ad741-5b63-4713-8c79-698219b308e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf56d53c-wphkc" Oct 30 23:29:17.316449 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:17.316428 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf56d53c-wphkc"] Oct 30 23:29:17.326847 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:17.319628 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" podUID="959b92af-9035-4dbd-aed3-5db4daaa63a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:17.326847 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:17.320423 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" Oct 30 23:29:17.353384 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:17.353353 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" Oct 30 23:29:17.387118 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:17.387086 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp"] Oct 30 23:29:17.391920 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:17.391881 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:29:17.392029 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:17.391975 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:29:17.392134 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:17.392059 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:29:17.590735 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:17.590707 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf56d53c-wphkc" Oct 30 23:29:17.704852 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:17.704815 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" event={"ID":"553c8b63-e975-498f-9313-878fbf99066d","Type":"ContainerStarted","Data":"0e539ca1c5acab406e0fcffffeb55994cadd84ab227f590dadf31a310a3c588a"} Oct 30 23:29:17.705003 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:17.704886 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" event={"ID":"553c8b63-e975-498f-9313-878fbf99066d","Type":"ContainerStarted","Data":"fa2c4d7f1f57261dc7f76d04e662b19ccecb9b7464c0822eb60234a443c61d3e"} Oct 30 23:29:17.798132 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:17.798084 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" Oct 30 23:29:17.826598 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:17.826545 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" podStartSLOduration=2.826530625 podStartE2EDuration="2.826530625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:29:17.823393625 +0000 UTC m=+1376.967223829" watchObservedRunningTime="2026-10-30 23:29:17.826530625 +0000 UTC m=+1376.970360878" Oct 30 23:29:18.000177 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:18.000080 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf56d53c-wphkc"] Oct 30 23:29:18.005764 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:29:18.005722 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode89ad741_5b63_4713_8c79_698219b308e0.slice/crio-df283ffe2384d318aeeb4d80f0b95d58b024e4598926f693f072d648b4e9614e WatchSource:0}: Error finding container df283ffe2384d318aeeb4d80f0b95d58b024e4598926f693f072d648b4e9614e: Status 404 returned error can't find the container with id df283ffe2384d318aeeb4d80f0b95d58b024e4598926f693f072d648b4e9614e Oct 30 23:29:18.312198 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:18.312121 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:18.555839 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:18.555593 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:18.715671 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:18.715631 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf56d53c-wphkc" event={"ID":"e89ad741-5b63-4713-8c79-698219b308e0","Type":"ContainerStarted","Data":"3fc139d4ffeea9682f0023467f174bedbaca0a60142b2be651a6e23e720dc534"} Oct 30 23:29:18.715867 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:18.715698 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf56d53c-wphkc" event={"ID":"e89ad741-5b63-4713-8c79-698219b308e0","Type":"ContainerStarted","Data":"df283ffe2384d318aeeb4d80f0b95d58b024e4598926f693f072d648b4e9614e"} Oct 30 23:29:18.799125 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:18.799094 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 23:29:18.799319 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:18.799150 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" podUID="553c8b63-e975-498f-9313-878fbf99066d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 23:29:18.840337 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:18.840291 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf56d53c-wphkc" podStartSLOduration=2.840276465 podStartE2EDuration="2.840276465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:29:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:29:18.838839961 +0000 UTC m=+1377.982670167" watchObservedRunningTime="2026-10-30 23:29:18.840276465 +0000 UTC m=+1377.984106670" Oct 30 23:29:19.725900 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:19.725866 4952 generic.go:356] "Generic (PLEG): container finished" podID="e89ad741-5b63-4713-8c79-698219b308e0" containerID="3fc139d4ffeea9682f0023467f174bedbaca0a60142b2be651a6e23e720dc534" exitCode=0 Oct 30 23:29:19.726323 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:19.725952 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf56d53c-wphkc" event={"ID":"e89ad741-5b63-4713-8c79-698219b308e0","Type":"ContainerDied","Data":"3fc139d4ffeea9682f0023467f174bedbaca0a60142b2be651a6e23e720dc534"} Oct 30 23:29:19.799083 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:19.799021 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 23:29:19.799381 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:19.799320 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" podUID="553c8b63-e975-498f-9313-878fbf99066d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 23:29:20.797768 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:20.797729 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 23:29:20.798286 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:20.797801 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" podUID="553c8b63-e975-498f-9313-878fbf99066d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 23:29:21.015849 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:21.015788 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf56d53c-wphkc" Oct 30 23:29:21.124785 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:21.124751 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e89ad741-5b63-4713-8c79-698219b308e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e89ad741-5b63-4713-8c79-698219b308e0-maas-api-openshift-ca\") pod \"e89ad741-5b63-4713-8c79-698219b308e0\" (UID: \"e89ad741-5b63-4713-8c79-698219b308e0\") " Oct 30 23:29:21.125027 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:21.125002 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e89ad741-5b63-4713-8c79-698219b308e0-kube-api-access-rwtst\" (UniqueName: \"kubernetes.io/projected/e89ad741-5b63-4713-8c79-698219b308e0-kube-api-access-rwtst\") pod \"e89ad741-5b63-4713-8c79-698219b308e0\" (UID: \"e89ad741-5b63-4713-8c79-698219b308e0\") " Oct 30 23:29:21.125210 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:21.125131 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e89ad741-5b63-4713-8c79-698219b308e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e89ad741-5b63-4713-8c79-698219b308e0-maas-api-xks-ca\") pod \"e89ad741-5b63-4713-8c79-698219b308e0\" (UID: \"e89ad741-5b63-4713-8c79-698219b308e0\") " Oct 30 23:29:21.125975 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:21.125949 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e89ad741-5b63-4713-8c79-698219b308e0-maas-api-openshift-ca" pod "e89ad741-5b63-4713-8c79-698219b308e0" (UID: "e89ad741-5b63-4713-8c79-698219b308e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:29:21.133120 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:21.128933 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e89ad741-5b63-4713-8c79-698219b308e0-maas-api-xks-ca" pod "e89ad741-5b63-4713-8c79-698219b308e0" (UID: "e89ad741-5b63-4713-8c79-698219b308e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:29:21.133120 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:21.130209 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e89ad741-5b63-4713-8c79-698219b308e0-kube-api-access-rwtst" pod "e89ad741-5b63-4713-8c79-698219b308e0" (UID: "e89ad741-5b63-4713-8c79-698219b308e0"). InnerVolumeSpecName "kube-api-access-rwtst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:29:21.151352 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:29:21.151320 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:29:21.227703 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:21.227673 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e89ad741-5b63-4713-8c79-698219b308e0-maas-api-openshift-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:29:21.227703 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:21.227699 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwtst\" (UniqueName: \"kubernetes.io/projected/e89ad741-5b63-4713-8c79-698219b308e0-kube-api-access-rwtst\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:29:21.227703 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:21.227709 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e89ad741-5b63-4713-8c79-698219b308e0-maas-api-xks-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:29:21.742733 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:21.742698 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf56d53c-wphkc" event={"ID":"e89ad741-5b63-4713-8c79-698219b308e0","Type":"ContainerDied","Data":"df283ffe2384d318aeeb4d80f0b95d58b024e4598926f693f072d648b4e9614e"} Oct 30 23:29:21.742733 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:21.742727 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf56d53c-wphkc" Oct 30 23:29:21.742927 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:21.742757 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df283ffe2384d318aeeb4d80f0b95d58b024e4598926f693f072d648b4e9614e" Oct 30 23:29:21.799399 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:21.799360 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 23:29:21.799872 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:21.799427 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" podUID="553c8b63-e975-498f-9313-878fbf99066d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 23:29:21.806428 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:21.806401 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:22.797812 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:22.797779 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 23:29:22.797987 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:22.797837 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" podUID="553c8b63-e975-498f-9313-878fbf99066d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 23:29:23.797956 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:23.797920 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 23:29:23.798381 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:23.797986 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" podUID="553c8b63-e975-498f-9313-878fbf99066d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 23:29:25.171658 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:25.171625 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" Oct 30 23:29:25.172186 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:25.172165 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" Oct 30 23:29:25.173608 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:25.173584 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" Oct 30 23:29:26.678225 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:26.678182 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:28.488673 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:28.488638 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:29.244023 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:29.243934 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:29.506636 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:29.506532 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:29.866671 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:29.866636 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:30.284703 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:30.284592 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:31.803018 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:31.802986 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:32.289571 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:32.289543 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:32.524540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:32.524512 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:32.799586 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:32.799550 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:33.063501 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:33.063425 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:35.461701 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:35.461665 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:36.291244 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:36.291207 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:36.611942 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:36.611903 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:36.855540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:36.855500 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:37.144799 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:37.144739 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:37.429953 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:37.428606 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:37.850226 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:37.850184 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:38.169218 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:38.169113 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:39.883300 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:39.883262 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:40.591451 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:40.591413 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:40.882441 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:40.882358 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:41.155176 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:41.154969 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:41.455909 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:41.455821 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:41.713440 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:41.713321 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:42.069195 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:42.069055 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:42.438582 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:42.438550 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t"] Oct 30 23:29:42.439143 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:42.438761 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" podUID="21849d16-2991-4d48-b38c-a3557eeaa97e" containerName="istio-proxy" containerID="cri-o://8ce37109a3a5d66f090a76470b8a76a809304109a30e8ab85074b80d1e8c7955" gracePeriod=30 Oct 30 23:29:42.490454 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:42.490420 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb"] Oct 30 23:29:42.490710 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:42.490684 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" podUID="ee837874-e627-482a-8b34-19c8c5b55fc5" containerName="istio-proxy" containerID="cri-o://048135fd65777bc3a84bdc765207f9e0ab6c430e3b478b8bdb224499745e7915" gracePeriod=30 Oct 30 23:29:42.783729 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:42.783629 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:43.960438 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:43.960402 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz"] Oct 30 23:29:43.961080 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:43.961063 4952 cpu_manager.go:414] "containerMap: removing container" podUID="e89ad741-5b63-4713-8c79-698219b308e0" containerName="revoke-keys" Oct 30 23:29:43.961080 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:43.961078 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e89ad741-5b63-4713-8c79-698219b308e0" containerName="revoke-keys" Oct 30 23:29:43.961256 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:43.961173 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="e89ad741-5b63-4713-8c79-698219b308e0" containerName="revoke-keys" Oct 30 23:29:43.964950 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:43.964928 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" Oct 30 23:29:44.024100 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:44.024070 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0646012c-55f4-4e2b-8dc8-dcd9603504a3-workload-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" Oct 30 23:29:44.024232 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:44.024146 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0646012c-55f4-4e2b-8dc8-dcd9603504a3-istio-envoy\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" Oct 30 23:29:44.024232 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:44.024178 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0646012c-55f4-4e2b-8dc8-dcd9603504a3-workload-certs\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" Oct 30 23:29:44.024232 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:44.024217 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0646012c-55f4-4e2b-8dc8-dcd9603504a3-istiod-ca-cert\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" Oct 30 23:29:44.024354 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:44.024285 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0646012c-55f4-4e2b-8dc8-dcd9603504a3-credential-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" Oct 30 23:29:44.024413 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:44.024396 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0646012c-55f4-4e2b-8dc8-dcd9603504a3-istio-podinfo\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" Oct 30 23:29:44.024460 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:44.024450 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0646012c-55f4-4e2b-8dc8-dcd9603504a3-istio-token\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" Oct 30 23:29:44.024551 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:44.024538 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0646012c-55f4-4e2b-8dc8-dcd9603504a3-istio-data\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" Oct 30 23:29:44.024592 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:44.024570 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd75n\" (UniqueName: \"kubernetes.io/projected/0646012c-55f4-4e2b-8dc8-dcd9603504a3-kube-api-access-jd75n\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" Oct 30 23:29:44.074070 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:44.071870 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz"] Oct 30 23:29:44.125708 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:44.125681 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0646012c-55f4-4e2b-8dc8-dcd9603504a3-istio-data\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" Oct 30 23:29:44.125876 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:44.125740 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd75n\" (UniqueName: \"kubernetes.io/projected/0646012c-55f4-4e2b-8dc8-dcd9603504a3-kube-api-access-jd75n\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" Oct 30 23:29:44.125876 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:44.125831 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0646012c-55f4-4e2b-8dc8-dcd9603504a3-workload-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" Oct 30 23:29:44.126189 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:44.126080 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0646012c-55f4-4e2b-8dc8-dcd9603504a3-istio-envoy\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" Oct 30 23:29:44.126189 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:44.126151 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0646012c-55f4-4e2b-8dc8-dcd9603504a3-workload-certs\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" Oct 30 23:29:44.126364 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:44.126193 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0646012c-55f4-4e2b-8dc8-dcd9603504a3-istio-data\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" Oct 30 23:29:44.126364 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:44.126210 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0646012c-55f4-4e2b-8dc8-dcd9603504a3-istiod-ca-cert\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" Oct 30 23:29:44.126364 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:44.126258 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0646012c-55f4-4e2b-8dc8-dcd9603504a3-credential-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" Oct 30 23:29:44.126517 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:44.126378 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0646012c-55f4-4e2b-8dc8-dcd9603504a3-istio-podinfo\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" Oct 30 23:29:44.126517 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:44.126445 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0646012c-55f4-4e2b-8dc8-dcd9603504a3-istio-token\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" Oct 30 23:29:44.126517 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:44.126460 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0646012c-55f4-4e2b-8dc8-dcd9603504a3-workload-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" Oct 30 23:29:44.126880 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:44.126745 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0646012c-55f4-4e2b-8dc8-dcd9603504a3-workload-certs\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" Oct 30 23:29:44.127191 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:44.127168 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0646012c-55f4-4e2b-8dc8-dcd9603504a3-credential-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" Oct 30 23:29:44.127366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:44.127187 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0646012c-55f4-4e2b-8dc8-dcd9603504a3-istiod-ca-cert\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" Oct 30 23:29:44.129054 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:44.129018 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0646012c-55f4-4e2b-8dc8-dcd9603504a3-istio-envoy\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" Oct 30 23:29:44.129797 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:44.129778 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0646012c-55f4-4e2b-8dc8-dcd9603504a3-istio-podinfo\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" Oct 30 23:29:44.183471 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:44.183443 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd75n\" (UniqueName: \"kubernetes.io/projected/0646012c-55f4-4e2b-8dc8-dcd9603504a3-kube-api-access-jd75n\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" Oct 30 23:29:44.183594 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:44.183569 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0646012c-55f4-4e2b-8dc8-dcd9603504a3-istio-token\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" Oct 30 23:29:44.282172 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:44.282094 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" Oct 30 23:29:44.721383 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:29:44.721352 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0646012c_55f4_4e2b_8dc8_dcd9603504a3.slice/crio-d77b0ea2a9ca13d08649fe225bcaa7895b61a1864e8720d45865b6a2c6d2b6a5 WatchSource:0}: Error finding container d77b0ea2a9ca13d08649fe225bcaa7895b61a1864e8720d45865b6a2c6d2b6a5: Status 404 returned error can't find the container with id d77b0ea2a9ca13d08649fe225bcaa7895b61a1864e8720d45865b6a2c6d2b6a5 Oct 30 23:29:44.721528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:44.721428 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz"] Oct 30 23:29:44.723954 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:44.723922 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:29:44.724126 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:44.723998 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:29:44.724126 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:44.724060 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:29:44.982941 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:44.982858 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" event={"ID":"0646012c-55f4-4e2b-8dc8-dcd9603504a3","Type":"ContainerStarted","Data":"d0c3fb9c24656cb620d1a370bb4b5a63286fcd56136944b52989b158225c6d40"} Oct 30 23:29:44.982941 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:44.982909 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" event={"ID":"0646012c-55f4-4e2b-8dc8-dcd9603504a3","Type":"ContainerStarted","Data":"d77b0ea2a9ca13d08649fe225bcaa7895b61a1864e8720d45865b6a2c6d2b6a5"} Oct 30 23:29:45.102929 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:45.102879 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" podStartSLOduration=2.102864635 podStartE2EDuration="2.102864635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:29:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:29:45.098927417 +0000 UTC m=+1404.242757623" watchObservedRunningTime="2026-10-30 23:29:45.102864635 +0000 UTC m=+1404.246694839" Oct 30 23:29:45.283083 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:45.282978 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" Oct 30 23:29:45.284447 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:45.284416 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 23:29:45.284570 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:45.284472 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" podUID="0646012c-55f4-4e2b-8dc8-dcd9603504a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 23:29:46.290499 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:46.290451 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 23:29:46.290997 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:46.290520 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" podUID="0646012c-55f4-4e2b-8dc8-dcd9603504a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 23:29:46.502314 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:46.502277 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:46.754151 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:46.754113 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:46.992187 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:46.992141 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:47.319747 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:47.318014 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 23:29:47.319747 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:47.318113 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" podUID="0646012c-55f4-4e2b-8dc8-dcd9603504a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 23:29:47.845651 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:47.845620 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" Oct 30 23:29:47.900060 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:47.897177 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21849d16-2991-4d48-b38c-a3557eeaa97e-istio-data" pod "21849d16-2991-4d48-b38c-a3557eeaa97e" (UID: "21849d16-2991-4d48-b38c-a3557eeaa97e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:29:47.900294 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:47.900272 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21849d16-2991-4d48-b38c-a3557eeaa97e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21849d16-2991-4d48-b38c-a3557eeaa97e-istio-data\") pod \"21849d16-2991-4d48-b38c-a3557eeaa97e\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " Oct 30 23:29:47.904507 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:47.901026 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21849d16-2991-4d48-b38c-a3557eeaa97e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21849d16-2991-4d48-b38c-a3557eeaa97e-workload-certs\") pod \"21849d16-2991-4d48-b38c-a3557eeaa97e\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " Oct 30 23:29:47.904507 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:47.901129 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21849d16-2991-4d48-b38c-a3557eeaa97e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21849d16-2991-4d48-b38c-a3557eeaa97e-istio-envoy\") pod \"21849d16-2991-4d48-b38c-a3557eeaa97e\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " Oct 30 23:29:47.904507 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:47.901259 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21849d16-2991-4d48-b38c-a3557eeaa97e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21849d16-2991-4d48-b38c-a3557eeaa97e-credential-socket\") pod \"21849d16-2991-4d48-b38c-a3557eeaa97e\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " Oct 30 23:29:47.904507 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:47.901314 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21849d16-2991-4d48-b38c-a3557eeaa97e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21849d16-2991-4d48-b38c-a3557eeaa97e-istiod-ca-cert\") pod \"21849d16-2991-4d48-b38c-a3557eeaa97e\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " Oct 30 23:29:47.904507 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:47.901397 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21849d16-2991-4d48-b38c-a3557eeaa97e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21849d16-2991-4d48-b38c-a3557eeaa97e-workload-socket\") pod \"21849d16-2991-4d48-b38c-a3557eeaa97e\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " Oct 30 23:29:47.904507 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:47.901433 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21849d16-2991-4d48-b38c-a3557eeaa97e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21849d16-2991-4d48-b38c-a3557eeaa97e-istio-podinfo\") pod \"21849d16-2991-4d48-b38c-a3557eeaa97e\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " Oct 30 23:29:47.904507 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:47.901485 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21849d16-2991-4d48-b38c-a3557eeaa97e-istio-token\" (UniqueName: \"kubernetes.io/projected/21849d16-2991-4d48-b38c-a3557eeaa97e-istio-token\") pod \"21849d16-2991-4d48-b38c-a3557eeaa97e\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " Oct 30 23:29:47.904507 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:47.901639 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21849d16-2991-4d48-b38c-a3557eeaa97e-kube-api-access-27ntl\" (UniqueName: \"kubernetes.io/projected/21849d16-2991-4d48-b38c-a3557eeaa97e-kube-api-access-27ntl\") pod \"21849d16-2991-4d48-b38c-a3557eeaa97e\" (UID: \"21849d16-2991-4d48-b38c-a3557eeaa97e\") " Oct 30 23:29:47.904507 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:47.902353 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21849d16-2991-4d48-b38c-a3557eeaa97e-istio-data\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:29:47.904507 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:47.903262 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21849d16-2991-4d48-b38c-a3557eeaa97e-istiod-ca-cert" pod "21849d16-2991-4d48-b38c-a3557eeaa97e" (UID: "21849d16-2991-4d48-b38c-a3557eeaa97e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:29:47.904507 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:47.903474 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21849d16-2991-4d48-b38c-a3557eeaa97e-workload-certs" pod "21849d16-2991-4d48-b38c-a3557eeaa97e" (UID: "21849d16-2991-4d48-b38c-a3557eeaa97e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:29:47.904507 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:47.903585 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21849d16-2991-4d48-b38c-a3557eeaa97e-workload-socket" pod "21849d16-2991-4d48-b38c-a3557eeaa97e" (UID: "21849d16-2991-4d48-b38c-a3557eeaa97e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:29:47.904507 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:47.903853 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21849d16-2991-4d48-b38c-a3557eeaa97e-credential-socket" pod "21849d16-2991-4d48-b38c-a3557eeaa97e" (UID: "21849d16-2991-4d48-b38c-a3557eeaa97e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:29:47.906433 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:47.906337 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21849d16-2991-4d48-b38c-a3557eeaa97e-istio-envoy" pod "21849d16-2991-4d48-b38c-a3557eeaa97e" (UID: "21849d16-2991-4d48-b38c-a3557eeaa97e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:29:47.906433 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:47.906389 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21849d16-2991-4d48-b38c-a3557eeaa97e-istio-podinfo" pod "21849d16-2991-4d48-b38c-a3557eeaa97e" (UID: "21849d16-2991-4d48-b38c-a3557eeaa97e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:29:47.908101 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:47.908024 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21849d16-2991-4d48-b38c-a3557eeaa97e-istio-token" pod "21849d16-2991-4d48-b38c-a3557eeaa97e" (UID: "21849d16-2991-4d48-b38c-a3557eeaa97e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:29:47.908304 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:47.908287 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21849d16-2991-4d48-b38c-a3557eeaa97e-kube-api-access-27ntl" pod "21849d16-2991-4d48-b38c-a3557eeaa97e" (UID: "21849d16-2991-4d48-b38c-a3557eeaa97e"). InnerVolumeSpecName "kube-api-access-27ntl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:29:47.908845 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:47.908764 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" Oct 30 23:29:48.003625 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.003596 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee837874-e627-482a-8b34-19c8c5b55fc5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee837874-e627-482a-8b34-19c8c5b55fc5-istiod-ca-cert\") pod \"ee837874-e627-482a-8b34-19c8c5b55fc5\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " Oct 30 23:29:48.003941 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.003921 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee837874-e627-482a-8b34-19c8c5b55fc5-istio-token\" (UniqueName: \"kubernetes.io/projected/ee837874-e627-482a-8b34-19c8c5b55fc5-istio-token\") pod \"ee837874-e627-482a-8b34-19c8c5b55fc5\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " Oct 30 23:29:48.004169 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.004153 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee837874-e627-482a-8b34-19c8c5b55fc5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee837874-e627-482a-8b34-19c8c5b55fc5-istio-podinfo\") pod \"ee837874-e627-482a-8b34-19c8c5b55fc5\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " Oct 30 23:29:48.004382 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.004366 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee837874-e627-482a-8b34-19c8c5b55fc5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee837874-e627-482a-8b34-19c8c5b55fc5-istio-data\") pod \"ee837874-e627-482a-8b34-19c8c5b55fc5\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " Oct 30 23:29:48.004595 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.004577 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee837874-e627-482a-8b34-19c8c5b55fc5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee837874-e627-482a-8b34-19c8c5b55fc5-credential-socket\") pod \"ee837874-e627-482a-8b34-19c8c5b55fc5\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " Oct 30 23:29:48.004751 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.004735 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee837874-e627-482a-8b34-19c8c5b55fc5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee837874-e627-482a-8b34-19c8c5b55fc5-istio-envoy\") pod \"ee837874-e627-482a-8b34-19c8c5b55fc5\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " Oct 30 23:29:48.004952 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.004936 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee837874-e627-482a-8b34-19c8c5b55fc5-kube-api-access-9mhn5\" (UniqueName: \"kubernetes.io/projected/ee837874-e627-482a-8b34-19c8c5b55fc5-kube-api-access-9mhn5\") pod \"ee837874-e627-482a-8b34-19c8c5b55fc5\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " Oct 30 23:29:48.006080 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.005099 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee837874-e627-482a-8b34-19c8c5b55fc5-credential-socket" pod "ee837874-e627-482a-8b34-19c8c5b55fc5" (UID: "ee837874-e627-482a-8b34-19c8c5b55fc5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:29:48.006080 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.005142 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee837874-e627-482a-8b34-19c8c5b55fc5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee837874-e627-482a-8b34-19c8c5b55fc5-workload-socket\") pod \"ee837874-e627-482a-8b34-19c8c5b55fc5\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " Oct 30 23:29:48.006080 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.005184 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee837874-e627-482a-8b34-19c8c5b55fc5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee837874-e627-482a-8b34-19c8c5b55fc5-workload-certs\") pod \"ee837874-e627-482a-8b34-19c8c5b55fc5\" (UID: \"ee837874-e627-482a-8b34-19c8c5b55fc5\") " Oct 30 23:29:48.006080 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.005195 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee837874-e627-482a-8b34-19c8c5b55fc5-istiod-ca-cert" pod "ee837874-e627-482a-8b34-19c8c5b55fc5" (UID: "ee837874-e627-482a-8b34-19c8c5b55fc5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:29:48.006080 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.006069 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee837874-e627-482a-8b34-19c8c5b55fc5-credential-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:29:48.006399 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.006109 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27ntl\" (UniqueName: \"kubernetes.io/projected/21849d16-2991-4d48-b38c-a3557eeaa97e-kube-api-access-27ntl\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:29:48.006399 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.006130 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21849d16-2991-4d48-b38c-a3557eeaa97e-workload-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:29:48.006399 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.006144 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21849d16-2991-4d48-b38c-a3557eeaa97e-istio-envoy\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:29:48.006399 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.006157 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21849d16-2991-4d48-b38c-a3557eeaa97e-credential-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:29:48.006399 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.006183 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21849d16-2991-4d48-b38c-a3557eeaa97e-istiod-ca-cert\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:29:48.006399 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.006197 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee837874-e627-482a-8b34-19c8c5b55fc5-istiod-ca-cert\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:29:48.006399 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.006211 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21849d16-2991-4d48-b38c-a3557eeaa97e-workload-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:29:48.006399 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.006224 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21849d16-2991-4d48-b38c-a3557eeaa97e-istio-podinfo\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:29:48.006399 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.006236 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21849d16-2991-4d48-b38c-a3557eeaa97e-istio-token\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:29:48.007366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.006686 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee837874-e627-482a-8b34-19c8c5b55fc5-workload-certs" pod "ee837874-e627-482a-8b34-19c8c5b55fc5" (UID: "ee837874-e627-482a-8b34-19c8c5b55fc5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:29:48.007366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.006963 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee837874-e627-482a-8b34-19c8c5b55fc5-workload-socket" pod "ee837874-e627-482a-8b34-19c8c5b55fc5" (UID: "ee837874-e627-482a-8b34-19c8c5b55fc5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:29:48.007563 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.007528 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee837874-e627-482a-8b34-19c8c5b55fc5-istio-data" pod "ee837874-e627-482a-8b34-19c8c5b55fc5" (UID: "ee837874-e627-482a-8b34-19c8c5b55fc5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:29:48.009450 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.009422 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee837874-e627-482a-8b34-19c8c5b55fc5-istio-podinfo" pod "ee837874-e627-482a-8b34-19c8c5b55fc5" (UID: "ee837874-e627-482a-8b34-19c8c5b55fc5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:29:48.009923 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.009881 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee837874-e627-482a-8b34-19c8c5b55fc5-kube-api-access-9mhn5" pod "ee837874-e627-482a-8b34-19c8c5b55fc5" (UID: "ee837874-e627-482a-8b34-19c8c5b55fc5"). InnerVolumeSpecName "kube-api-access-9mhn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:29:48.011470 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.011437 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee837874-e627-482a-8b34-19c8c5b55fc5-istio-token" pod "ee837874-e627-482a-8b34-19c8c5b55fc5" (UID: "ee837874-e627-482a-8b34-19c8c5b55fc5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:29:48.011916 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.011892 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee837874-e627-482a-8b34-19c8c5b55fc5-istio-envoy" pod "ee837874-e627-482a-8b34-19c8c5b55fc5" (UID: "ee837874-e627-482a-8b34-19c8c5b55fc5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:29:48.025567 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.025538 4952 generic.go:356] "Generic (PLEG): container finished" podID="ee837874-e627-482a-8b34-19c8c5b55fc5" containerID="048135fd65777bc3a84bdc765207f9e0ab6c430e3b478b8bdb224499745e7915" exitCode=0 Oct 30 23:29:48.025709 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.025688 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" event={"ID":"ee837874-e627-482a-8b34-19c8c5b55fc5","Type":"ContainerDied","Data":"048135fd65777bc3a84bdc765207f9e0ab6c430e3b478b8bdb224499745e7915"} Oct 30 23:29:48.025757 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.025748 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" event={"ID":"ee837874-e627-482a-8b34-19c8c5b55fc5","Type":"ContainerDied","Data":"c19abf1dd8fb76c704805f6048759f06fa0e8f7a779deae6d5c1bda49caf5768"} Oct 30 23:29:48.025792 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.025776 4952 scope.go:122] "RemoveContainer" containerID="048135fd65777bc3a84bdc765207f9e0ab6c430e3b478b8bdb224499745e7915" Oct 30 23:29:48.025936 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.025922 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb" Oct 30 23:29:48.028738 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.028712 4952 generic.go:356] "Generic (PLEG): container finished" podID="21849d16-2991-4d48-b38c-a3557eeaa97e" containerID="8ce37109a3a5d66f090a76470b8a76a809304109a30e8ab85074b80d1e8c7955" exitCode=0 Oct 30 23:29:48.028844 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.028781 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" event={"ID":"21849d16-2991-4d48-b38c-a3557eeaa97e","Type":"ContainerDied","Data":"8ce37109a3a5d66f090a76470b8a76a809304109a30e8ab85074b80d1e8c7955"} Oct 30 23:29:48.028844 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.028818 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" event={"ID":"21849d16-2991-4d48-b38c-a3557eeaa97e","Type":"ContainerDied","Data":"9fcddd1832d28a82eee9afa4bc5c43498ddc99087096059d9e63fd8c774f95b0"} Oct 30 23:29:48.028967 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.028882 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t" Oct 30 23:29:48.042707 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.042599 4952 scope.go:122] "RemoveContainer" containerID="048135fd65777bc3a84bdc765207f9e0ab6c430e3b478b8bdb224499745e7915" Oct 30 23:29:48.042933 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:29:48.042908 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"048135fd65777bc3a84bdc765207f9e0ab6c430e3b478b8bdb224499745e7915\": container with ID starting with 048135fd65777bc3a84bdc765207f9e0ab6c430e3b478b8bdb224499745e7915 not found: ID does not exist" containerID="048135fd65777bc3a84bdc765207f9e0ab6c430e3b478b8bdb224499745e7915" Oct 30 23:29:48.043004 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.042944 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"048135fd65777bc3a84bdc765207f9e0ab6c430e3b478b8bdb224499745e7915"} err="failed to get container status \"048135fd65777bc3a84bdc765207f9e0ab6c430e3b478b8bdb224499745e7915\": rpc error: code = NotFound desc = could not find container \"048135fd65777bc3a84bdc765207f9e0ab6c430e3b478b8bdb224499745e7915\": container with ID starting with 048135fd65777bc3a84bdc765207f9e0ab6c430e3b478b8bdb224499745e7915 not found: ID does not exist" Oct 30 23:29:48.043004 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.042967 4952 scope.go:122] "RemoveContainer" containerID="8ce37109a3a5d66f090a76470b8a76a809304109a30e8ab85074b80d1e8c7955" Oct 30 23:29:48.057287 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.057189 4952 scope.go:122] "RemoveContainer" containerID="8ce37109a3a5d66f090a76470b8a76a809304109a30e8ab85074b80d1e8c7955" Oct 30 23:29:48.057604 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:29:48.057582 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ce37109a3a5d66f090a76470b8a76a809304109a30e8ab85074b80d1e8c7955\": container with ID starting with 8ce37109a3a5d66f090a76470b8a76a809304109a30e8ab85074b80d1e8c7955 not found: ID does not exist" containerID="8ce37109a3a5d66f090a76470b8a76a809304109a30e8ab85074b80d1e8c7955" Oct 30 23:29:48.057701 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.057608 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ce37109a3a5d66f090a76470b8a76a809304109a30e8ab85074b80d1e8c7955"} err="failed to get container status \"8ce37109a3a5d66f090a76470b8a76a809304109a30e8ab85074b80d1e8c7955\": rpc error: code = NotFound desc = could not find container \"8ce37109a3a5d66f090a76470b8a76a809304109a30e8ab85074b80d1e8c7955\": container with ID starting with 8ce37109a3a5d66f090a76470b8a76a809304109a30e8ab85074b80d1e8c7955 not found: ID does not exist" Oct 30 23:29:48.110658 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.110630 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee837874-e627-482a-8b34-19c8c5b55fc5-istio-token\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:29:48.110924 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.110904 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee837874-e627-482a-8b34-19c8c5b55fc5-istio-podinfo\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:29:48.111082 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.111067 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee837874-e627-482a-8b34-19c8c5b55fc5-istio-data\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:29:48.111192 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.111180 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee837874-e627-482a-8b34-19c8c5b55fc5-istio-envoy\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:29:48.111291 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.111277 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mhn5\" (UniqueName: \"kubernetes.io/projected/ee837874-e627-482a-8b34-19c8c5b55fc5-kube-api-access-9mhn5\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:29:48.111381 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.111370 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee837874-e627-482a-8b34-19c8c5b55fc5-workload-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:29:48.111504 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.111490 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee837874-e627-482a-8b34-19c8c5b55fc5-workload-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:29:48.153262 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.153171 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:48.302079 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.285872 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb"] Oct 30 23:29:48.302784 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.302755 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 23:29:48.303100 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.303015 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" podUID="0646012c-55f4-4e2b-8dc8-dcd9603504a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 23:29:48.309613 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.309588 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6zvgb"] Oct 30 23:29:48.407114 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.406830 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:48.409082 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.408958 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t"] Oct 30 23:29:48.414214 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.414186 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-69s4t"] Oct 30 23:29:48.744286 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.744205 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:48.998941 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.998862 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21849d16-2991-4d48-b38c-a3557eeaa97e" path="/var/lib/kubelet/pods/21849d16-2991-4d48-b38c-a3557eeaa97e/volumes" Oct 30 23:29:48.999452 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:48.999435 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee837874-e627-482a-8b34-19c8c5b55fc5" path="/var/lib/kubelet/pods/ee837874-e627-482a-8b34-19c8c5b55fc5/volumes" Oct 30 23:29:49.287120 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:49.283194 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 23:29:49.287120 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:49.283250 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" podUID="0646012c-55f4-4e2b-8dc8-dcd9603504a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 23:29:50.289383 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:50.289349 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 23:29:50.289923 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:50.289891 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" podUID="0646012c-55f4-4e2b-8dc8-dcd9603504a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 23:29:50.802147 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:50.802113 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9"] Oct 30 23:29:50.802718 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:50.802705 4952 cpu_manager.go:414] "containerMap: removing container" podUID="21849d16-2991-4d48-b38c-a3557eeaa97e" containerName="istio-proxy" Oct 30 23:29:50.802774 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:50.802720 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21849d16-2991-4d48-b38c-a3557eeaa97e" containerName="istio-proxy" Oct 30 23:29:50.802774 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:50.802758 4952 cpu_manager.go:414] "containerMap: removing container" podUID="ee837874-e627-482a-8b34-19c8c5b55fc5" containerName="istio-proxy" Oct 30 23:29:50.802774 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:50.802764 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee837874-e627-482a-8b34-19c8c5b55fc5" containerName="istio-proxy" Oct 30 23:29:50.802871 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:50.802858 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="21849d16-2991-4d48-b38c-a3557eeaa97e" containerName="istio-proxy" Oct 30 23:29:50.802904 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:50.802873 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee837874-e627-482a-8b34-19c8c5b55fc5" containerName="istio-proxy" Oct 30 23:29:50.808137 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:50.808118 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9" Oct 30 23:29:50.849990 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:50.849961 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t42s9\" (UID: \"305e64da-56e7-4fbb-b9fa-a7f486ed1dac\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9" Oct 30 23:29:50.850165 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:50.850005 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t42s9\" (UID: \"305e64da-56e7-4fbb-b9fa-a7f486ed1dac\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9" Oct 30 23:29:50.850165 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:50.850140 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t42s9\" (UID: \"305e64da-56e7-4fbb-b9fa-a7f486ed1dac\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9" Oct 30 23:29:50.850165 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:50.850159 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2lms\" (UniqueName: \"kubernetes.io/projected/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-kube-api-access-j2lms\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t42s9\" (UID: \"305e64da-56e7-4fbb-b9fa-a7f486ed1dac\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9" Oct 30 23:29:50.850334 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:50.850186 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t42s9\" (UID: \"305e64da-56e7-4fbb-b9fa-a7f486ed1dac\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9" Oct 30 23:29:50.850392 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:50.850368 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t42s9\" (UID: \"305e64da-56e7-4fbb-b9fa-a7f486ed1dac\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9" Oct 30 23:29:50.850451 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:50.850412 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t42s9\" (UID: \"305e64da-56e7-4fbb-b9fa-a7f486ed1dac\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9" Oct 30 23:29:50.860304 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:50.860269 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 23:29:50.860304 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:50.860290 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 23:29:50.860304 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:50.860291 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 23:29:50.860572 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:50.860550 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-fdjft\"" Oct 30 23:29:50.916275 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:50.916245 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9"] Oct 30 23:29:50.951986 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:50.951954 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t42s9\" (UID: \"305e64da-56e7-4fbb-b9fa-a7f486ed1dac\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9" Oct 30 23:29:50.952187 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:50.951999 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2lms\" (UniqueName: \"kubernetes.io/projected/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-kube-api-access-j2lms\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t42s9\" (UID: \"305e64da-56e7-4fbb-b9fa-a7f486ed1dac\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9" Oct 30 23:29:50.952187 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:50.952173 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t42s9\" (UID: \"305e64da-56e7-4fbb-b9fa-a7f486ed1dac\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9" Oct 30 23:29:50.952428 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:50.952402 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t42s9\" (UID: \"305e64da-56e7-4fbb-b9fa-a7f486ed1dac\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9" Oct 30 23:29:50.952491 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:50.952456 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t42s9\" (UID: \"305e64da-56e7-4fbb-b9fa-a7f486ed1dac\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9" Oct 30 23:29:50.952915 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:50.952603 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t42s9\" (UID: \"305e64da-56e7-4fbb-b9fa-a7f486ed1dac\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9" Oct 30 23:29:50.953088 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:50.952916 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t42s9\" (UID: \"305e64da-56e7-4fbb-b9fa-a7f486ed1dac\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9" Oct 30 23:29:50.953747 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:50.953717 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t42s9\" (UID: \"305e64da-56e7-4fbb-b9fa-a7f486ed1dac\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9" Oct 30 23:29:50.956076 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:50.955532 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t42s9\" (UID: \"305e64da-56e7-4fbb-b9fa-a7f486ed1dac\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9" Oct 30 23:29:50.956076 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:50.955840 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t42s9\" (UID: \"305e64da-56e7-4fbb-b9fa-a7f486ed1dac\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9" Oct 30 23:29:50.956076 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:50.956018 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t42s9\" (UID: \"305e64da-56e7-4fbb-b9fa-a7f486ed1dac\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9" Oct 30 23:29:50.959352 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:50.959332 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t42s9\" (UID: \"305e64da-56e7-4fbb-b9fa-a7f486ed1dac\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9" Oct 30 23:29:51.009286 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:51.009256 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2lms\" (UniqueName: \"kubernetes.io/projected/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-kube-api-access-j2lms\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t42s9\" (UID: \"305e64da-56e7-4fbb-b9fa-a7f486ed1dac\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9" Oct 30 23:29:51.009985 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:51.009952 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t42s9\" (UID: \"305e64da-56e7-4fbb-b9fa-a7f486ed1dac\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9" Oct 30 23:29:51.121434 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:51.121401 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9" Oct 30 23:29:51.149362 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:29:51.149328 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:29:51.282685 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:51.282649 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 23:29:51.282935 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:51.282710 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" podUID="0646012c-55f4-4e2b-8dc8-dcd9603504a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 23:29:51.298999 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:51.298967 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:29:51.577409 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:29:51.577381 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod305e64da_56e7_4fbb_b9fa_a7f486ed1dac.slice/crio-91f342561b62ca8a7f32085baf617fb326d752d05613c39b711389456d53c104 WatchSource:0}: Error finding container 91f342561b62ca8a7f32085baf617fb326d752d05613c39b711389456d53c104: Status 404 returned error can't find the container with id 91f342561b62ca8a7f32085baf617fb326d752d05613c39b711389456d53c104 Oct 30 23:29:51.578341 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:51.578317 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9"] Oct 30 23:29:51.579378 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:51.579361 4952 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 23:29:52.120551 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:52.120518 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9" event={"ID":"305e64da-56e7-4fbb-b9fa-a7f486ed1dac","Type":"ContainerStarted","Data":"91f342561b62ca8a7f32085baf617fb326d752d05613c39b711389456d53c104"} Oct 30 23:29:53.202073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:53.199140 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" podUID="86a57a21-72bc-4ce4-8d1e-bf29214ad4de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:53.332070 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:53.330577 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" podUID="959b92af-9035-4dbd-aed3-5db4daaa63a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:53.381553 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:53.366987 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:29:53.381553 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:53.367068 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" podUID="0646012c-55f4-4e2b-8dc8-dcd9603504a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:53.409458 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:53.408875 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" podUID="86ec338e-ad13-4029-a1cc-88650c6a27ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded" Oct 30 23:29:53.587153 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:53.586542 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" podUID="44adfd6a-2626-4378-bca1-d46ee1670b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:53.688065 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:53.678684 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" podUID="709f007e-0ccc-40f5-816e-ed24702ac107" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:53.704092 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:53.700135 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" podUID="da4935dd-c148-4cc4-84ab-1bfd91cce9b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:53.704092 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:53.700526 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" podUID="86ec338e-ad13-4029-a1cc-88650c6a27ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 23:29:53.752066 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:53.749574 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" podUID="6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:53.759182 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:53.759136 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" podUID="28913940-90f7-4d80-89b7-f5eece975d02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:53.883133 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:53.880453 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" podUID="e662ea1f-f4ae-43ff-852c-ed3d6d282c74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:53.883133 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:53.881580 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" Oct 30 23:29:53.887384 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:53.884243 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" Oct 30 23:29:53.913555 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:53.912099 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" Oct 30 23:29:54.003162 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:54.003116 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" podUID="43e39353-ae85-4d76-9900-7e7927409098" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:54.094176 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:54.094137 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:29:54.094458 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:54.094430 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" podUID="ec9e7fff-1e25-4f16-ab95-f1a1d65448da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:54.250119 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:54.244715 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" podUID="da4935dd-c148-4cc4-84ab-1bfd91cce9b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:54.250119 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:54.246192 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" podUID="86a57a21-72bc-4ce4-8d1e-bf29214ad4de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 30 23:29:54.259100 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:54.247029 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" Oct 30 23:29:54.332152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:54.330128 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" podUID="ab5286a5-6020-4c0b-9511-db63d4472a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 23:29:54.332152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:54.331021 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" podUID="959b92af-9035-4dbd-aed3-5db4daaa63a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:54.332152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:54.331394 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" Oct 30 23:29:54.403500 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:54.403405 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:54.599123 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:54.598099 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" podUID="44adfd6a-2626-4378-bca1-d46ee1670b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 23:29:54.603208 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:54.603179 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" Oct 30 23:29:54.715064 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:54.714542 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" podUID="709f007e-0ccc-40f5-816e-ed24702ac107" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:54.732347 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:54.732318 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" Oct 30 23:29:54.733134 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:54.733096 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" podUID="86ec338e-ad13-4029-a1cc-88650c6a27ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:54.779063 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:54.767386 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" podUID="6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:54.779063 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:54.768148 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" Oct 30 23:29:54.779063 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:54.768218 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" Oct 30 23:29:54.779063 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:54.768772 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" podUID="28913940-90f7-4d80-89b7-f5eece975d02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:54.779063 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:54.769865 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" Oct 30 23:29:54.817076 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:54.799447 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" podUID="da4935dd-c148-4cc4-84ab-1bfd91cce9b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:54.817076 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:54.799694 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" Oct 30 23:29:55.027414 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:55.014400 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" podUID="43e39353-ae85-4d76-9900-7e7927409098" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:55.027414 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:55.015506 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" podUID="145e7c01-efec-46e6-a748-56b39de76f83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 23:29:55.027414 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:55.016605 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" podUID="e662ea1f-f4ae-43ff-852c-ed3d6d282c74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 30 23:29:55.027414 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:55.018248 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" Oct 30 23:29:55.027414 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:55.018309 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" Oct 30 23:29:55.027414 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:55.019625 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" podUID="44adfd6a-2626-4378-bca1-d46ee1670b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:55.128107 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:55.127117 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" podUID="86a57a21-72bc-4ce4-8d1e-bf29214ad4de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:55.287422 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:55.281726 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" podUID="86a57a21-72bc-4ce4-8d1e-bf29214ad4de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 30 23:29:55.294472 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:55.294414 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" Oct 30 23:29:55.351842 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:55.339647 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" podUID="959b92af-9035-4dbd-aed3-5db4daaa63a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:55.372272 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:55.356321 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" podUID="ab5286a5-6020-4c0b-9511-db63d4472a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 23:29:55.372272 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:55.357353 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" Oct 30 23:29:55.612066 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:55.608409 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" podUID="44adfd6a-2626-4378-bca1-d46ee1670b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 23:29:55.704061 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:55.703573 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:55.745116 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:55.741151 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" podUID="86ec338e-ad13-4029-a1cc-88650c6a27ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:55.771153 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:55.771111 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" podUID="6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 23:29:55.784187 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:55.784150 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" podUID="28913940-90f7-4d80-89b7-f5eece975d02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:55.802171 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:55.802129 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" podUID="da4935dd-c148-4cc4-84ab-1bfd91cce9b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:56.038539 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:56.038433 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" podUID="145e7c01-efec-46e6-a748-56b39de76f83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 23:29:56.039222 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:56.039176 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" podUID="43e39353-ae85-4d76-9900-7e7927409098" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 23:29:56.428914 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:56.039965 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" Oct 30 23:29:56.445831 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:56.092615 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:56.446173 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:56.293174 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" podUID="86a57a21-72bc-4ce4-8d1e-bf29214ad4de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:56.463183 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:56.293213 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" podUID="709f007e-0ccc-40f5-816e-ed24702ac107" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:56.480132 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:56.356024 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" podUID="959b92af-9035-4dbd-aed3-5db4daaa63a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:56.480720 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:56.363283 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" podUID="ab5286a5-6020-4c0b-9511-db63d4472a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:56.616069 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:56.613985 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" podUID="44adfd6a-2626-4378-bca1-d46ee1670b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:56.692022 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:56.691919 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" podUID="e662ea1f-f4ae-43ff-852c-ed3d6d282c74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:56.803196 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:56.756128 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" podUID="86ec338e-ad13-4029-a1cc-88650c6a27ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:56.803196 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:56.798152 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" podUID="28913940-90f7-4d80-89b7-f5eece975d02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:56.803196 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:56.798555 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" podUID="6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 23:29:56.812448 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:56.812410 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" podUID="da4935dd-c148-4cc4-84ab-1bfd91cce9b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 23:29:57.097128 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:57.095182 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" podUID="145e7c01-efec-46e6-a748-56b39de76f83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 23:29:57.564267 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:57.562291 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" podUID="86a57a21-72bc-4ce4-8d1e-bf29214ad4de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Oct 30 23:29:57.626193 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:57.592262 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" podUID="43e39353-ae85-4d76-9900-7e7927409098" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:57.711124 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:57.653626 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" podUID="ab5286a5-6020-4c0b-9511-db63d4472a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded" Oct 30 23:29:57.711124 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:57.654158 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" podUID="44adfd6a-2626-4378-bca1-d46ee1670b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 23:29:57.711124 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:57.654584 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" podUID="ab5286a5-6020-4c0b-9511-db63d4472a0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:57.711124 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:57.654811 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" podUID="709f007e-0ccc-40f5-816e-ed24702ac107" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Oct 30 23:29:57.711124 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:57.655102 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" podUID="959b92af-9035-4dbd-aed3-5db4daaa63a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 23:29:57.711124 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:57.655394 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" Oct 30 23:29:57.784151 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:57.755276 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" podUID="e662ea1f-f4ae-43ff-852c-ed3d6d282c74" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:57.811137 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:57.811093 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" podUID="6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:57.859446 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:57.820866 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" podUID="28913940-90f7-4d80-89b7-f5eece975d02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 30 23:29:57.859446 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:57.836974 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" Oct 30 23:29:58.136884 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:58.122120 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" podUID="145e7c01-efec-46e6-a748-56b39de76f83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:58.162061 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:58.156709 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" podUID="709f007e-0ccc-40f5-816e-ed24702ac107" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": context deadline exceeded" Oct 30 23:29:58.452201 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:58.269695 4952 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" podUID="145e7c01-efec-46e6-a748-56b39de76f83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:58.452201 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:58.269800 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" podUID="da4935dd-c148-4cc4-84ab-1bfd91cce9b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 23:29:58.584255 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:58.584208 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" podUID="86a57a21-72bc-4ce4-8d1e-bf29214ad4de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:58.605076 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:58.597614 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" podUID="43e39353-ae85-4d76-9900-7e7927409098" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:58.677067 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:58.677008 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" podUID="959b92af-9035-4dbd-aed3-5db4daaa63a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:58.715936 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:58.715854 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" podUID="44adfd6a-2626-4378-bca1-d46ee1670b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:58.734099 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:58.705202 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" podUID="709f007e-0ccc-40f5-816e-ed24702ac107" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:29:58.735561 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:58.735532 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" Oct 30 23:29:58.736636 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:58.736609 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" Oct 30 23:29:58.736763 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:58.736649 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" Oct 30 23:29:58.736763 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:58.736696 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" Oct 30 23:29:58.736763 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:58.736735 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" Oct 30 23:29:58.736920 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:58.736780 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" Oct 30 23:29:58.758893 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:58.758868 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" Oct 30 23:29:58.760330 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:58.760250 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" Oct 30 23:29:58.780500 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:58.780474 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:58.843533 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:58.843508 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" Oct 30 23:29:58.848225 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:58.848078 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" Oct 30 23:29:58.848225 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:58.848134 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" Oct 30 23:29:59.391239 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:59.391173 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:29:59.775298 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:29:59.775205 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:30:00.039624 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:00.039551 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:30:00.202413 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:00.202377 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh"] Oct 30 23:30:00.221066 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:00.218999 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh" Oct 30 23:30:00.264553 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:00.264518 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548"] Oct 30 23:30:00.288406 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:00.285964 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\"" Oct 30 23:30:00.310108 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:00.297365 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpnt9\" (UniqueName: \"kubernetes.io/projected/b92369f1-9c4d-4654-91c8-4c8925596b07-kube-api-access-qpnt9\") pod \"maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh\" (UID: \"b92369f1-9c4d-4654-91c8-4c8925596b07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh" Oct 30 23:30:00.310108 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:00.308919 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b92369f1-9c4d-4654-91c8-4c8925596b07-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh\" (UID: \"b92369f1-9c4d-4654-91c8-4c8925596b07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh" Oct 30 23:30:00.310108 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:00.307569 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548" Oct 30 23:30:00.318348 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:00.307484 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f"] Oct 30 23:30:00.340422 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:00.340379 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh"] Oct 30 23:30:00.340562 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:00.340467 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548"] Oct 30 23:30:00.340686 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:00.340662 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f" Oct 30 23:30:00.386770 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:00.386730 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f"] Oct 30 23:30:00.412807 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:00.412759 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpnt9\" (UniqueName: \"kubernetes.io/projected/b92369f1-9c4d-4654-91c8-4c8925596b07-kube-api-access-qpnt9\") pod \"maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh\" (UID: \"b92369f1-9c4d-4654-91c8-4c8925596b07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh" Oct 30 23:30:00.412990 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:00.412880 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b92369f1-9c4d-4654-91c8-4c8925596b07-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh\" (UID: \"b92369f1-9c4d-4654-91c8-4c8925596b07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh" Oct 30 23:30:00.412990 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:00.412932 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwcns\" (UniqueName: \"kubernetes.io/projected/36cdec6c-538d-4983-ab87-83a1ff62a222-kube-api-access-nwcns\") pod \"maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548\" (UID: \"36cdec6c-538d-4983-ab87-83a1ff62a222\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548" Oct 30 23:30:00.413172 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:00.413115 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9m2q\" (UniqueName: \"kubernetes.io/projected/84968a4b-45c2-4631-a9ae-2480b2ffbdda-kube-api-access-c9m2q\") pod \"maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f\" (UID: \"84968a4b-45c2-4631-a9ae-2480b2ffbdda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f" Oct 30 23:30:00.413303 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:00.413261 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84968a4b-45c2-4631-a9ae-2480b2ffbdda-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f\" (UID: \"84968a4b-45c2-4631-a9ae-2480b2ffbdda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f" Oct 30 23:30:00.413397 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:00.413378 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36cdec6c-538d-4983-ab87-83a1ff62a222-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548\" (UID: \"36cdec6c-538d-4983-ab87-83a1ff62a222\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548" Oct 30 23:30:00.413638 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:00.413617 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b92369f1-9c4d-4654-91c8-4c8925596b07-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh\" (UID: \"b92369f1-9c4d-4654-91c8-4c8925596b07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh" Oct 30 23:30:00.488376 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:00.488323 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpnt9\" (UniqueName: \"kubernetes.io/projected/b92369f1-9c4d-4654-91c8-4c8925596b07-kube-api-access-qpnt9\") pod \"maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh\" (UID: \"b92369f1-9c4d-4654-91c8-4c8925596b07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh" Oct 30 23:30:00.514940 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:00.514910 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36cdec6c-538d-4983-ab87-83a1ff62a222-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548\" (UID: \"36cdec6c-538d-4983-ab87-83a1ff62a222\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548" Oct 30 23:30:00.515125 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:00.515076 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwcns\" (UniqueName: \"kubernetes.io/projected/36cdec6c-538d-4983-ab87-83a1ff62a222-kube-api-access-nwcns\") pod \"maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548\" (UID: \"36cdec6c-538d-4983-ab87-83a1ff62a222\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548" Oct 30 23:30:00.515207 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:00.515191 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9m2q\" (UniqueName: \"kubernetes.io/projected/84968a4b-45c2-4631-a9ae-2480b2ffbdda-kube-api-access-c9m2q\") pod \"maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f\" (UID: \"84968a4b-45c2-4631-a9ae-2480b2ffbdda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f" Oct 30 23:30:00.515273 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:00.515251 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84968a4b-45c2-4631-a9ae-2480b2ffbdda-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f\" (UID: \"84968a4b-45c2-4631-a9ae-2480b2ffbdda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f" Oct 30 23:30:00.515813 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:00.515781 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36cdec6c-538d-4983-ab87-83a1ff62a222-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548\" (UID: \"36cdec6c-538d-4983-ab87-83a1ff62a222\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548" Oct 30 23:30:00.515967 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:00.515948 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84968a4b-45c2-4631-a9ae-2480b2ffbdda-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f\" (UID: \"84968a4b-45c2-4631-a9ae-2480b2ffbdda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f" Oct 30 23:30:00.520935 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:00.520901 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:30:00.570570 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:00.570286 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh" Oct 30 23:30:00.572963 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:00.572934 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9m2q\" (UniqueName: \"kubernetes.io/projected/84968a4b-45c2-4631-a9ae-2480b2ffbdda-kube-api-access-c9m2q\") pod \"maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f\" (UID: \"84968a4b-45c2-4631-a9ae-2480b2ffbdda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f" Oct 30 23:30:00.574432 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:00.574364 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwcns\" (UniqueName: \"kubernetes.io/projected/36cdec6c-538d-4983-ab87-83a1ff62a222-kube-api-access-nwcns\") pod \"maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548\" (UID: \"36cdec6c-538d-4983-ab87-83a1ff62a222\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548" Oct 30 23:30:00.687062 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:00.685454 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548" Oct 30 23:30:00.692226 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:00.687659 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f" Oct 30 23:30:01.252299 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:01.252261 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh"] Oct 30 23:30:01.276636 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:30:01.276608 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb92369f1_9c4d_4654_91c8_4c8925596b07.slice/crio-89b9ff00dcd6941ea751965101db8cc93b9f9937c356a6697b649dfabf9c58a2 WatchSource:0}: Error finding container 89b9ff00dcd6941ea751965101db8cc93b9f9937c356a6697b649dfabf9c58a2: Status 404 returned error can't find the container with id 89b9ff00dcd6941ea751965101db8cc93b9f9937c356a6697b649dfabf9c58a2 Oct 30 23:30:01.302575 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:01.302552 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548"] Oct 30 23:30:01.308696 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:30:01.308594 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84968a4b_45c2_4631_a9ae_2480b2ffbdda.slice/crio-77dcd5904f8bacd24e04a8953b883e5cd3d769a7a2fb00520c73507f2603a7bb WatchSource:0}: Error finding container 77dcd5904f8bacd24e04a8953b883e5cd3d769a7a2fb00520c73507f2603a7bb: Status 404 returned error can't find the container with id 77dcd5904f8bacd24e04a8953b883e5cd3d769a7a2fb00520c73507f2603a7bb Oct 30 23:30:01.310064 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:01.309892 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f"] Oct 30 23:30:01.313165 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:30:01.313144 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36cdec6c_538d_4983_ab87_83a1ff62a222.slice/crio-4cfe564e190786bb0e48c2644df8d7126b7182a96372d0d5b103c5bab21a9119 WatchSource:0}: Error finding container 4cfe564e190786bb0e48c2644df8d7126b7182a96372d0d5b103c5bab21a9119: Status 404 returned error can't find the container with id 4cfe564e190786bb0e48c2644df8d7126b7182a96372d0d5b103c5bab21a9119 Oct 30 23:30:01.403077 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:01.403021 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh" event={"ID":"b92369f1-9c4d-4654-91c8-4c8925596b07","Type":"ContainerStarted","Data":"89b9ff00dcd6941ea751965101db8cc93b9f9937c356a6697b649dfabf9c58a2"} Oct 30 23:30:01.404812 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:01.404787 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548" event={"ID":"36cdec6c-538d-4983-ab87-83a1ff62a222","Type":"ContainerStarted","Data":"4cfe564e190786bb0e48c2644df8d7126b7182a96372d0d5b103c5bab21a9119"} Oct 30 23:30:01.407650 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:01.407625 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f" event={"ID":"84968a4b-45c2-4631-a9ae-2480b2ffbdda","Type":"ContainerStarted","Data":"77dcd5904f8bacd24e04a8953b883e5cd3d769a7a2fb00520c73507f2603a7bb"} Oct 30 23:30:02.441058 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:02.440567 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh" event={"ID":"b92369f1-9c4d-4654-91c8-4c8925596b07","Type":"ContainerStarted","Data":"975a5243d5675d6ccccbb9304c30b7929366e50761e324b7f29ea67bbda5ba1a"} Oct 30 23:30:02.452127 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:02.449812 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548" event={"ID":"36cdec6c-538d-4983-ab87-83a1ff62a222","Type":"ContainerStarted","Data":"5910815fc0c286a3fd8151c0275416f2f01ee59d6af5cb9b9d4ad5a08177d513"} Oct 30 23:30:02.455063 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:02.452807 4952 generic.go:356] "Generic (PLEG): container finished" podID="84968a4b-45c2-4631-a9ae-2480b2ffbdda" containerID="66434b4e3f17ec5b036a71dfd4019b74c674a1515c280db6880510e5facd1650" exitCode=60 Oct 30 23:30:02.455063 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:02.453077 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f" event={"ID":"84968a4b-45c2-4631-a9ae-2480b2ffbdda","Type":"ContainerDied","Data":"66434b4e3f17ec5b036a71dfd4019b74c674a1515c280db6880510e5facd1650"} Oct 30 23:30:02.455063 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:02.453447 4952 scope.go:122] "RemoveContainer" containerID="66434b4e3f17ec5b036a71dfd4019b74c674a1515c280db6880510e5facd1650" Oct 30 23:30:02.556792 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:02.556728 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh" podStartSLOduration=2.5567107460000003 podStartE2EDuration="2.556710746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:30:02.555586996 +0000 UTC m=+1421.699417237" watchObservedRunningTime="2026-10-30 23:30:02.556710746 +0000 UTC m=+1421.700540953" Oct 30 23:30:02.793826 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:02.793767 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548" podStartSLOduration=2.793747718 podStartE2EDuration="2.793747718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:30:02.790376046 +0000 UTC m=+1421.934206279" watchObservedRunningTime="2026-10-30 23:30:02.793747718 +0000 UTC m=+1421.937577925" Oct 30 23:30:03.273788 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:03.273752 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:30:03.273896 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:03.273818 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:30:03.464524 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:03.464489 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9" event={"ID":"305e64da-56e7-4fbb-b9fa-a7f486ed1dac","Type":"ContainerStarted","Data":"c6479aece7c37b72c59d5cb18e908a14dd685036306922b4892afc31bb8e0a35"} Oct 30 23:30:03.464942 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:03.464559 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9" Oct 30 23:30:03.466413 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:03.466386 4952 generic.go:356] "Generic (PLEG): container finished" podID="b92369f1-9c4d-4654-91c8-4c8925596b07" containerID="975a5243d5675d6ccccbb9304c30b7929366e50761e324b7f29ea67bbda5ba1a" exitCode=60 Oct 30 23:30:03.466547 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:03.466474 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh" event={"ID":"b92369f1-9c4d-4654-91c8-4c8925596b07","Type":"ContainerDied","Data":"975a5243d5675d6ccccbb9304c30b7929366e50761e324b7f29ea67bbda5ba1a"} Oct 30 23:30:03.466754 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:03.466735 4952 scope.go:122] "RemoveContainer" containerID="975a5243d5675d6ccccbb9304c30b7929366e50761e324b7f29ea67bbda5ba1a" Oct 30 23:30:03.468481 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:03.468447 4952 generic.go:356] "Generic (PLEG): container finished" podID="36cdec6c-538d-4983-ab87-83a1ff62a222" containerID="5910815fc0c286a3fd8151c0275416f2f01ee59d6af5cb9b9d4ad5a08177d513" exitCode=60 Oct 30 23:30:03.468587 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:03.468522 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548" event={"ID":"36cdec6c-538d-4983-ab87-83a1ff62a222","Type":"ContainerDied","Data":"5910815fc0c286a3fd8151c0275416f2f01ee59d6af5cb9b9d4ad5a08177d513"} Oct 30 23:30:03.468793 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:03.468777 4952 scope.go:122] "RemoveContainer" containerID="5910815fc0c286a3fd8151c0275416f2f01ee59d6af5cb9b9d4ad5a08177d513" Oct 30 23:30:03.471317 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:03.471137 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f" event={"ID":"84968a4b-45c2-4631-a9ae-2480b2ffbdda","Type":"ContainerStarted","Data":"2741632dabf2a88b8a1bd54a873e6b9a68ad6fa7ea0b1049d222147dbc697a22"} Oct 30 23:30:03.582277 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:03.582215 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9" podStartSLOduration=1.8893235320000001 podStartE2EDuration="13.582195217s" totalImagesPullingTime="11.692871685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:29:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:30:03.579880923 +0000 UTC m=+1422.723711146" watchObservedRunningTime="2026-10-30 23:30:03.582195217 +0000 UTC m=+1422.726025423" Oct 30 23:30:03.815872 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:03.815762 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f" podStartSLOduration=3.815747979 podStartE2EDuration="3.815747979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:30:03.810593276 +0000 UTC m=+1422.954423483" watchObservedRunningTime="2026-10-30 23:30:03.815747979 +0000 UTC m=+1422.959578184" Oct 30 23:30:04.503429 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:04.503398 4952 scope.go:122] "RemoveContainer" containerID="2741632dabf2a88b8a1bd54a873e6b9a68ad6fa7ea0b1049d222147dbc697a22" Oct 30 23:30:04.504408 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:30:04.504373 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f_odh-ai-gateway-infra(84968a4b-45c2-4631-a9ae-2480b2ffbdda)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f" podUID="84968a4b-45c2-4631-a9ae-2480b2ffbdda" Oct 30 23:30:04.506112 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:04.502616 4952 generic.go:356] "Generic (PLEG): container finished" podID="84968a4b-45c2-4631-a9ae-2480b2ffbdda" containerID="2741632dabf2a88b8a1bd54a873e6b9a68ad6fa7ea0b1049d222147dbc697a22" exitCode=60 Oct 30 23:30:04.506410 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:04.503270 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f" event={"ID":"84968a4b-45c2-4631-a9ae-2480b2ffbdda","Type":"ContainerDied","Data":"2741632dabf2a88b8a1bd54a873e6b9a68ad6fa7ea0b1049d222147dbc697a22"} Oct 30 23:30:04.506667 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:04.506644 4952 scope.go:122] "RemoveContainer" containerID="66434b4e3f17ec5b036a71dfd4019b74c674a1515c280db6880510e5facd1650" Oct 30 23:30:04.511415 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:04.511388 4952 generic.go:356] "Generic (PLEG): container finished" podID="b92369f1-9c4d-4654-91c8-4c8925596b07" containerID="3909eacf6f515a3e4aa91102ca55ded2aa93bc577d32747d947bc418564f164a" exitCode=60 Oct 30 23:30:04.511656 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:04.511473 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh" event={"ID":"b92369f1-9c4d-4654-91c8-4c8925596b07","Type":"ContainerDied","Data":"3909eacf6f515a3e4aa91102ca55ded2aa93bc577d32747d947bc418564f164a"} Oct 30 23:30:04.512471 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:04.512452 4952 scope.go:122] "RemoveContainer" containerID="3909eacf6f515a3e4aa91102ca55ded2aa93bc577d32747d947bc418564f164a" Oct 30 23:30:04.512890 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:30:04.512867 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh_odh-ai-gateway-infra(b92369f1-9c4d-4654-91c8-4c8925596b07)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh" podUID="b92369f1-9c4d-4654-91c8-4c8925596b07" Oct 30 23:30:04.514528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:04.514501 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548" event={"ID":"36cdec6c-538d-4983-ab87-83a1ff62a222","Type":"ContainerStarted","Data":"263c75d261b981c0a5faebf683741483c6e09c19e96e4f2b09063904bacde283"} Oct 30 23:30:04.517265 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:04.517240 4952 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-t42s9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 23:30:04.517441 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:04.517415 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9" podUID="305e64da-56e7-4fbb-b9fa-a7f486ed1dac" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:30:04.533520 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:04.533487 4952 scope.go:122] "RemoveContainer" containerID="975a5243d5675d6ccccbb9304c30b7929366e50761e324b7f29ea67bbda5ba1a" Oct 30 23:30:05.045506 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:05.045404 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:30:05.288967 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:05.288927 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:30:05.546168 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:05.546138 4952 scope.go:122] "RemoveContainer" containerID="3909eacf6f515a3e4aa91102ca55ded2aa93bc577d32747d947bc418564f164a" Oct 30 23:30:05.546626 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:30:05.546520 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh_odh-ai-gateway-infra(b92369f1-9c4d-4654-91c8-4c8925596b07)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh" podUID="b92369f1-9c4d-4654-91c8-4c8925596b07" Oct 30 23:30:05.547951 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:05.547917 4952 generic.go:356] "Generic (PLEG): container finished" podID="36cdec6c-538d-4983-ab87-83a1ff62a222" containerID="263c75d261b981c0a5faebf683741483c6e09c19e96e4f2b09063904bacde283" exitCode=60 Oct 30 23:30:05.548140 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:05.547997 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548" event={"ID":"36cdec6c-538d-4983-ab87-83a1ff62a222","Type":"ContainerDied","Data":"263c75d261b981c0a5faebf683741483c6e09c19e96e4f2b09063904bacde283"} Oct 30 23:30:05.548140 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:05.548067 4952 scope.go:122] "RemoveContainer" containerID="5910815fc0c286a3fd8151c0275416f2f01ee59d6af5cb9b9d4ad5a08177d513" Oct 30 23:30:05.548324 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:05.548304 4952 scope.go:122] "RemoveContainer" containerID="263c75d261b981c0a5faebf683741483c6e09c19e96e4f2b09063904bacde283" Oct 30 23:30:05.548642 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:30:05.548620 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548_odh-ai-gateway-infra(36cdec6c-538d-4983-ab87-83a1ff62a222)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548" podUID="36cdec6c-538d-4983-ab87-83a1ff62a222" Oct 30 23:30:05.550209 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:05.550193 4952 scope.go:122] "RemoveContainer" containerID="2741632dabf2a88b8a1bd54a873e6b9a68ad6fa7ea0b1049d222147dbc697a22" Oct 30 23:30:05.550459 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:30:05.550436 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f_odh-ai-gateway-infra(84968a4b-45c2-4631-a9ae-2480b2ffbdda)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f" podUID="84968a4b-45c2-4631-a9ae-2480b2ffbdda" Oct 30 23:30:05.559350 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:05.559327 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:30:05.809615 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:05.809513 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:30:06.127952 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:06.127914 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:30:06.563148 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:06.563065 4952 scope.go:122] "RemoveContainer" containerID="263c75d261b981c0a5faebf683741483c6e09c19e96e4f2b09063904bacde283" Oct 30 23:30:06.563628 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:30:06.563457 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548_odh-ai-gateway-infra(36cdec6c-538d-4983-ab87-83a1ff62a222)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548" podUID="36cdec6c-538d-4983-ab87-83a1ff62a222" Oct 30 23:30:07.515725 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:07.515697 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9" Oct 30 23:30:11.188133 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:11.188091 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:30:12.706977 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:12.706941 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g"] Oct 30 23:30:12.711658 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:12.711642 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" Oct 30 23:30:12.794498 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:12.794467 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g"] Oct 30 23:30:12.799745 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:12.799719 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53756ab3-37fe-4d52-b197-7066f9b257c1-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-9ml7g\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" Oct 30 23:30:12.799907 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:12.799886 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53756ab3-37fe-4d52-b197-7066f9b257c1-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-9ml7g\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" Oct 30 23:30:12.799952 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:12.799928 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53756ab3-37fe-4d52-b197-7066f9b257c1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-9ml7g\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" Oct 30 23:30:12.800073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:12.800015 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzcgb\" (UniqueName: \"kubernetes.io/projected/53756ab3-37fe-4d52-b197-7066f9b257c1-kube-api-access-jzcgb\") pod \"maas-default-gateway-openshift-default-5df48fcc99-9ml7g\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" Oct 30 23:30:12.800127 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:12.800108 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53756ab3-37fe-4d52-b197-7066f9b257c1-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-9ml7g\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" Oct 30 23:30:12.800204 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:12.800189 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53756ab3-37fe-4d52-b197-7066f9b257c1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-9ml7g\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" Oct 30 23:30:12.800279 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:12.800261 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53756ab3-37fe-4d52-b197-7066f9b257c1-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-9ml7g\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" Oct 30 23:30:12.800342 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:12.800321 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53756ab3-37fe-4d52-b197-7066f9b257c1-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-9ml7g\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" Oct 30 23:30:12.800382 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:12.800355 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53756ab3-37fe-4d52-b197-7066f9b257c1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-9ml7g\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" Oct 30 23:30:12.901204 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:12.901171 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzcgb\" (UniqueName: \"kubernetes.io/projected/53756ab3-37fe-4d52-b197-7066f9b257c1-kube-api-access-jzcgb\") pod \"maas-default-gateway-openshift-default-5df48fcc99-9ml7g\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" Oct 30 23:30:12.901366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:12.901231 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53756ab3-37fe-4d52-b197-7066f9b257c1-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-9ml7g\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" Oct 30 23:30:12.901366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:12.901309 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53756ab3-37fe-4d52-b197-7066f9b257c1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-9ml7g\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" Oct 30 23:30:12.901487 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:12.901370 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53756ab3-37fe-4d52-b197-7066f9b257c1-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-9ml7g\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" Oct 30 23:30:12.901487 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:12.901420 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53756ab3-37fe-4d52-b197-7066f9b257c1-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-9ml7g\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" Oct 30 23:30:12.901487 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:12.901452 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53756ab3-37fe-4d52-b197-7066f9b257c1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-9ml7g\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" Oct 30 23:30:12.901639 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:12.901510 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53756ab3-37fe-4d52-b197-7066f9b257c1-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-9ml7g\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" Oct 30 23:30:12.901698 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:12.901634 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53756ab3-37fe-4d52-b197-7066f9b257c1-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-9ml7g\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" Oct 30 23:30:12.901698 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:12.901688 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53756ab3-37fe-4d52-b197-7066f9b257c1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-9ml7g\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" Oct 30 23:30:12.901817 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:12.901752 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53756ab3-37fe-4d52-b197-7066f9b257c1-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-9ml7g\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" Oct 30 23:30:12.901873 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:12.901825 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53756ab3-37fe-4d52-b197-7066f9b257c1-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-9ml7g\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" Oct 30 23:30:12.901983 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:12.901957 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53756ab3-37fe-4d52-b197-7066f9b257c1-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-9ml7g\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" Oct 30 23:30:12.902306 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:12.902083 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53756ab3-37fe-4d52-b197-7066f9b257c1-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-9ml7g\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" Oct 30 23:30:12.902869 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:12.902842 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53756ab3-37fe-4d52-b197-7066f9b257c1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-9ml7g\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" Oct 30 23:30:12.903796 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:12.903773 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53756ab3-37fe-4d52-b197-7066f9b257c1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-9ml7g\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" Oct 30 23:30:12.904182 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:12.904166 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53756ab3-37fe-4d52-b197-7066f9b257c1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-9ml7g\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" Oct 30 23:30:12.959741 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:12.959624 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzcgb\" (UniqueName: \"kubernetes.io/projected/53756ab3-37fe-4d52-b197-7066f9b257c1-kube-api-access-jzcgb\") pod \"maas-default-gateway-openshift-default-5df48fcc99-9ml7g\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" Oct 30 23:30:12.961150 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:12.961130 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53756ab3-37fe-4d52-b197-7066f9b257c1-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-9ml7g\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" Oct 30 23:30:13.025643 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:13.025618 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" Oct 30 23:30:13.470713 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:30:13.470681 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53756ab3_37fe_4d52_b197_7066f9b257c1.slice/crio-aa9b43ed53597f476fc0598b656ebcccc998354daa2ff4fc9fc85f916a22f6ba WatchSource:0}: Error finding container aa9b43ed53597f476fc0598b656ebcccc998354daa2ff4fc9fc85f916a22f6ba: Status 404 returned error can't find the container with id aa9b43ed53597f476fc0598b656ebcccc998354daa2ff4fc9fc85f916a22f6ba Oct 30 23:30:13.471245 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:13.471223 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g"] Oct 30 23:30:13.473307 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:13.473277 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:30:13.473404 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:13.473340 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:30:13.473404 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:13.473367 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:30:13.628062 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:13.624783 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" event={"ID":"53756ab3-37fe-4d52-b197-7066f9b257c1","Type":"ContainerStarted","Data":"5ad7201c57b5e2cf62b3e2ea872c54c88482f26b5ab5b9d18dd0d91e9cb98f63"} Oct 30 23:30:13.628062 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:13.624843 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" event={"ID":"53756ab3-37fe-4d52-b197-7066f9b257c1","Type":"ContainerStarted","Data":"aa9b43ed53597f476fc0598b656ebcccc998354daa2ff4fc9fc85f916a22f6ba"} Oct 30 23:30:13.747502 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:13.747394 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" podStartSLOduration=1.747380992 podStartE2EDuration="1.747380992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:30:13.74263997 +0000 UTC m=+1432.886470221" watchObservedRunningTime="2026-10-30 23:30:13.747380992 +0000 UTC m=+1432.891211259" Oct 30 23:30:14.032614 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:14.032310 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" Oct 30 23:30:14.034071 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:14.034012 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-9ml7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 23:30:14.034213 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:14.034084 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" podUID="53756ab3-37fe-4d52-b197-7066f9b257c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 23:30:15.026765 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:15.026729 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-9ml7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 23:30:15.027186 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:15.026781 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" podUID="53756ab3-37fe-4d52-b197-7066f9b257c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 23:30:15.645788 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:15.645753 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686"] Oct 30 23:30:15.650308 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:15.650285 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686" Oct 30 23:30:15.703322 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:15.703298 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-49d07c83-serving-cert\"" Oct 30 23:30:15.708146 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:15.708112 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686"] Oct 30 23:30:15.744463 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:15.744422 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4863fa66-8c17-49c8-be19-01693e905cd3-metrics-certs\") pod \"maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686\" (UID: \"4863fa66-8c17-49c8-be19-01693e905cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686" Oct 30 23:30:15.744605 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:15.744558 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl7ww\" (UniqueName: \"kubernetes.io/projected/4863fa66-8c17-49c8-be19-01693e905cd3-kube-api-access-jl7ww\") pod \"maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686\" (UID: \"4863fa66-8c17-49c8-be19-01693e905cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686" Oct 30 23:30:15.744674 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:15.744616 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4863fa66-8c17-49c8-be19-01693e905cd3-openshift-service-ca\") pod \"maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686\" (UID: \"4863fa66-8c17-49c8-be19-01693e905cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686" Oct 30 23:30:15.744903 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:15.744883 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4863fa66-8c17-49c8-be19-01693e905cd3-maas-api-tls\") pod \"maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686\" (UID: \"4863fa66-8c17-49c8-be19-01693e905cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686" Oct 30 23:30:15.846757 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:15.846721 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4863fa66-8c17-49c8-be19-01693e905cd3-metrics-certs\") pod \"maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686\" (UID: \"4863fa66-8c17-49c8-be19-01693e905cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686" Oct 30 23:30:15.846920 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:15.846775 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl7ww\" (UniqueName: \"kubernetes.io/projected/4863fa66-8c17-49c8-be19-01693e905cd3-kube-api-access-jl7ww\") pod \"maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686\" (UID: \"4863fa66-8c17-49c8-be19-01693e905cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686" Oct 30 23:30:15.846920 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:15.846801 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4863fa66-8c17-49c8-be19-01693e905cd3-openshift-service-ca\") pod \"maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686\" (UID: \"4863fa66-8c17-49c8-be19-01693e905cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686" Oct 30 23:30:15.847173 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:15.847149 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4863fa66-8c17-49c8-be19-01693e905cd3-maas-api-tls\") pod \"maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686\" (UID: \"4863fa66-8c17-49c8-be19-01693e905cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686" Oct 30 23:30:15.847280 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:30:15.847255 4952 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-49d07c83-serving-cert: secret "maas-api-e2e-ait-49d07c83-serving-cert" not found Oct 30 23:30:15.847372 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:30:15.847352 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4863fa66-8c17-49c8-be19-01693e905cd3-maas-api-tls podName:4863fa66-8c17-49c8-be19-01693e905cd3 nodeName:}" failed. No retries permitted until 2026-10-30 23:30:16.34732972 +0000 UTC m=+1435.491159917 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4863fa66-8c17-49c8-be19-01693e905cd3-maas-api-tls") pod "maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686" (UID: "4863fa66-8c17-49c8-be19-01693e905cd3") : secret "maas-api-e2e-ait-49d07c83-serving-cert" not found Oct 30 23:30:15.847652 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:15.847633 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4863fa66-8c17-49c8-be19-01693e905cd3-openshift-service-ca\") pod \"maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686\" (UID: \"4863fa66-8c17-49c8-be19-01693e905cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686" Oct 30 23:30:15.849484 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:15.849460 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4863fa66-8c17-49c8-be19-01693e905cd3-metrics-certs\") pod \"maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686\" (UID: \"4863fa66-8c17-49c8-be19-01693e905cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686" Oct 30 23:30:15.904850 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:15.904778 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl7ww\" (UniqueName: \"kubernetes.io/projected/4863fa66-8c17-49c8-be19-01693e905cd3-kube-api-access-jl7ww\") pod \"maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686\" (UID: \"4863fa66-8c17-49c8-be19-01693e905cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686" Oct 30 23:30:15.931307 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:15.931270 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:30:16.026954 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:16.026924 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-9ml7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 23:30:16.027380 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:16.026982 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" podUID="53756ab3-37fe-4d52-b197-7066f9b257c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 23:30:16.155360 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:16.155283 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:30:16.356366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:16.356333 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4863fa66-8c17-49c8-be19-01693e905cd3-maas-api-tls\") pod \"maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686\" (UID: \"4863fa66-8c17-49c8-be19-01693e905cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686" Oct 30 23:30:16.359783 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:16.359755 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4863fa66-8c17-49c8-be19-01693e905cd3-maas-api-tls\") pod \"maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686\" (UID: \"4863fa66-8c17-49c8-be19-01693e905cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686" Oct 30 23:30:16.385689 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:16.385653 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:30:16.565004 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:16.564926 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686" Oct 30 23:30:16.963221 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:16.963188 4952 scope.go:122] "RemoveContainer" containerID="2741632dabf2a88b8a1bd54a873e6b9a68ad6fa7ea0b1049d222147dbc697a22" Oct 30 23:30:17.002612 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:17.002575 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686"] Oct 30 23:30:17.004514 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:30:17.004489 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4863fa66_8c17_49c8_be19_01693e905cd3.slice/crio-f84e37e9b1b57b8155ca13f1b8cb741937241837ac900fe712bd472999f8e28e WatchSource:0}: Error finding container f84e37e9b1b57b8155ca13f1b8cb741937241837ac900fe712bd472999f8e28e: Status 404 returned error can't find the container with id f84e37e9b1b57b8155ca13f1b8cb741937241837ac900fe712bd472999f8e28e Oct 30 23:30:17.026847 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:17.026820 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-9ml7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 23:30:17.026955 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:17.026874 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" podUID="53756ab3-37fe-4d52-b197-7066f9b257c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 23:30:17.692114 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:17.691597 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686" event={"ID":"4863fa66-8c17-49c8-be19-01693e905cd3","Type":"ContainerStarted","Data":"f84e37e9b1b57b8155ca13f1b8cb741937241837ac900fe712bd472999f8e28e"} Oct 30 23:30:17.697350 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:17.697316 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f" event={"ID":"84968a4b-45c2-4631-a9ae-2480b2ffbdda","Type":"ContainerStarted","Data":"6ae372fdabcc8d0a7326167b836b4c80637202ed261cf85f12507466cb9578f3"} Oct 30 23:30:18.026532 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:18.026502 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-9ml7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 23:30:18.026681 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:18.026559 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" podUID="53756ab3-37fe-4d52-b197-7066f9b257c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 23:30:18.662293 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:18.662261 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:30:18.708687 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:18.708650 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686" event={"ID":"4863fa66-8c17-49c8-be19-01693e905cd3","Type":"ContainerStarted","Data":"eeb0dae4ce8033e253176b7f39eddc47cf4672548d80a20d7e9385a547bb0340"} Oct 30 23:30:18.709228 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:18.709143 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686" Oct 30 23:30:18.711489 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:18.711466 4952 generic.go:356] "Generic (PLEG): container finished" podID="84968a4b-45c2-4631-a9ae-2480b2ffbdda" containerID="6ae372fdabcc8d0a7326167b836b4c80637202ed261cf85f12507466cb9578f3" exitCode=60 Oct 30 23:30:18.711645 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:18.711520 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f" event={"ID":"84968a4b-45c2-4631-a9ae-2480b2ffbdda","Type":"ContainerDied","Data":"6ae372fdabcc8d0a7326167b836b4c80637202ed261cf85f12507466cb9578f3"} Oct 30 23:30:18.711645 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:18.711562 4952 scope.go:122] "RemoveContainer" containerID="2741632dabf2a88b8a1bd54a873e6b9a68ad6fa7ea0b1049d222147dbc697a22" Oct 30 23:30:18.711969 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:18.711951 4952 scope.go:122] "RemoveContainer" containerID="6ae372fdabcc8d0a7326167b836b4c80637202ed261cf85f12507466cb9578f3" Oct 30 23:30:18.712356 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:30:18.712333 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f_odh-ai-gateway-infra(84968a4b-45c2-4631-a9ae-2480b2ffbdda)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f" podUID="84968a4b-45c2-4631-a9ae-2480b2ffbdda" Oct 30 23:30:18.826130 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:18.826091 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f"] Oct 30 23:30:18.889469 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:18.883833 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686" podStartSLOduration=2.935454007 podStartE2EDuration="3.883812343s" totalImagesPullingTime="948.358336ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:30:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:30:18.882622026 +0000 UTC m=+1438.026452233" watchObservedRunningTime="2026-10-30 23:30:18.883812343 +0000 UTC m=+1438.027642549" Oct 30 23:30:18.985129 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:18.966479 4952 scope.go:122] "RemoveContainer" containerID="3909eacf6f515a3e4aa91102ca55ded2aa93bc577d32747d947bc418564f164a" Oct 30 23:30:19.001097 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:18.989589 4952 scope.go:122] "RemoveContainer" containerID="263c75d261b981c0a5faebf683741483c6e09c19e96e4f2b09063904bacde283" Oct 30 23:30:19.025887 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:19.025862 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-9ml7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 23:30:19.026118 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:19.026083 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" podUID="53756ab3-37fe-4d52-b197-7066f9b257c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 23:30:19.693111 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:19.693074 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:30:19.810782 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:19.810739 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh" event={"ID":"b92369f1-9c4d-4654-91c8-4c8925596b07","Type":"ContainerStarted","Data":"4c2a87dd66c954908d8d12bdf74e609df45c04b0b6f27996e747ba0b966f60ea"} Oct 30 23:30:19.816259 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:19.815485 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548" event={"ID":"36cdec6c-538d-4983-ab87-83a1ff62a222","Type":"ContainerStarted","Data":"e6d91669d0abdf0adfc7992c4d1328e08629db5bc125d33dea90b0e7c6ba6c09"} Oct 30 23:30:19.978157 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:19.975220 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:30:20.026582 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:20.026554 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-9ml7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 23:30:20.026705 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:20.026609 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" podUID="53756ab3-37fe-4d52-b197-7066f9b257c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 23:30:20.044927 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:20.044904 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f" Oct 30 23:30:20.233166 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:20.230116 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84968a4b-45c2-4631-a9ae-2480b2ffbdda-kube-api-access-c9m2q\" (UniqueName: \"kubernetes.io/projected/84968a4b-45c2-4631-a9ae-2480b2ffbdda-kube-api-access-c9m2q\") pod \"84968a4b-45c2-4631-a9ae-2480b2ffbdda\" (UID: \"84968a4b-45c2-4631-a9ae-2480b2ffbdda\") " Oct 30 23:30:20.233166 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:20.230350 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84968a4b-45c2-4631-a9ae-2480b2ffbdda-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84968a4b-45c2-4631-a9ae-2480b2ffbdda-maas-api-service-ca\") pod \"84968a4b-45c2-4631-a9ae-2480b2ffbdda\" (UID: \"84968a4b-45c2-4631-a9ae-2480b2ffbdda\") " Oct 30 23:30:20.371755 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:20.371562 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84968a4b-45c2-4631-a9ae-2480b2ffbdda-maas-api-service-ca" pod "84968a4b-45c2-4631-a9ae-2480b2ffbdda" (UID: "84968a4b-45c2-4631-a9ae-2480b2ffbdda"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:30:20.389448 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:20.389410 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84968a4b-45c2-4631-a9ae-2480b2ffbdda-kube-api-access-c9m2q" pod "84968a4b-45c2-4631-a9ae-2480b2ffbdda" (UID: "84968a4b-45c2-4631-a9ae-2480b2ffbdda"). InnerVolumeSpecName "kube-api-access-c9m2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:30:20.469124 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:20.465269 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84968a4b-45c2-4631-a9ae-2480b2ffbdda-maas-api-service-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:30:20.469124 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:20.465306 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9m2q\" (UniqueName: \"kubernetes.io/projected/84968a4b-45c2-4631-a9ae-2480b2ffbdda-kube-api-access-c9m2q\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:30:20.828864 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:20.828835 4952 generic.go:356] "Generic (PLEG): container finished" podID="b92369f1-9c4d-4654-91c8-4c8925596b07" containerID="4c2a87dd66c954908d8d12bdf74e609df45c04b0b6f27996e747ba0b966f60ea" exitCode=60 Oct 30 23:30:20.829394 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:20.828913 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh" event={"ID":"b92369f1-9c4d-4654-91c8-4c8925596b07","Type":"ContainerDied","Data":"4c2a87dd66c954908d8d12bdf74e609df45c04b0b6f27996e747ba0b966f60ea"} Oct 30 23:30:20.829394 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:20.829009 4952 scope.go:122] "RemoveContainer" containerID="3909eacf6f515a3e4aa91102ca55ded2aa93bc577d32747d947bc418564f164a" Oct 30 23:30:20.829546 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:20.829400 4952 scope.go:122] "RemoveContainer" containerID="4c2a87dd66c954908d8d12bdf74e609df45c04b0b6f27996e747ba0b966f60ea" Oct 30 23:30:20.829787 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:30:20.829758 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh_odh-ai-gateway-infra(b92369f1-9c4d-4654-91c8-4c8925596b07)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh" podUID="b92369f1-9c4d-4654-91c8-4c8925596b07" Oct 30 23:30:20.831305 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:20.831281 4952 generic.go:356] "Generic (PLEG): container finished" podID="36cdec6c-538d-4983-ab87-83a1ff62a222" containerID="e6d91669d0abdf0adfc7992c4d1328e08629db5bc125d33dea90b0e7c6ba6c09" exitCode=60 Oct 30 23:30:20.831407 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:20.831351 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548" event={"ID":"36cdec6c-538d-4983-ab87-83a1ff62a222","Type":"ContainerDied","Data":"e6d91669d0abdf0adfc7992c4d1328e08629db5bc125d33dea90b0e7c6ba6c09"} Oct 30 23:30:20.831738 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:20.831721 4952 scope.go:122] "RemoveContainer" containerID="e6d91669d0abdf0adfc7992c4d1328e08629db5bc125d33dea90b0e7c6ba6c09" Oct 30 23:30:20.832023 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:30:20.832005 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548_odh-ai-gateway-infra(36cdec6c-538d-4983-ab87-83a1ff62a222)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548" podUID="36cdec6c-538d-4983-ab87-83a1ff62a222" Oct 30 23:30:20.832964 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:20.832943 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f" event={"ID":"84968a4b-45c2-4631-a9ae-2480b2ffbdda","Type":"ContainerDied","Data":"77dcd5904f8bacd24e04a8953b883e5cd3d769a7a2fb00520c73507f2603a7bb"} Oct 30 23:30:20.832964 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:20.832960 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f" Oct 30 23:30:20.848914 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:20.848888 4952 scope.go:122] "RemoveContainer" containerID="263c75d261b981c0a5faebf683741483c6e09c19e96e4f2b09063904bacde283" Oct 30 23:30:20.865920 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:20.865899 4952 scope.go:122] "RemoveContainer" containerID="6ae372fdabcc8d0a7326167b836b4c80637202ed261cf85f12507466cb9578f3" Oct 30 23:30:20.937426 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:20.937394 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh"] Oct 30 23:30:21.026800 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:21.026766 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-9ml7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 23:30:21.026961 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:21.026811 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" podUID="53756ab3-37fe-4d52-b197-7066f9b257c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 23:30:21.062998 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:21.062967 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548"] Oct 30 23:30:21.160780 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:30:21.160743 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:30:21.176730 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:21.176696 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:30:21.218784 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:21.218750 4952 scope.go:122] "RemoveContainer" containerID="5918aa8982e370c1213a5cdd870149a91a1c42e01a1e6fa073d33277ad275092" Oct 30 23:30:21.275622 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:21.273683 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f"] Oct 30 23:30:21.279401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:21.279373 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890050-brl7f"] Oct 30 23:30:22.119603 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:22.119544 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh" Oct 30 23:30:22.121809 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:22.121785 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548" Oct 30 23:30:22.299127 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:22.299095 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36cdec6c-538d-4983-ab87-83a1ff62a222-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36cdec6c-538d-4983-ab87-83a1ff62a222-maas-api-service-ca\") pod \"36cdec6c-538d-4983-ab87-83a1ff62a222\" (UID: \"36cdec6c-538d-4983-ab87-83a1ff62a222\") " Oct 30 23:30:22.299325 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:22.299229 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36cdec6c-538d-4983-ab87-83a1ff62a222-kube-api-access-nwcns\" (UniqueName: \"kubernetes.io/projected/36cdec6c-538d-4983-ab87-83a1ff62a222-kube-api-access-nwcns\") pod \"36cdec6c-538d-4983-ab87-83a1ff62a222\" (UID: \"36cdec6c-538d-4983-ab87-83a1ff62a222\") " Oct 30 23:30:22.299325 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:22.299274 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b92369f1-9c4d-4654-91c8-4c8925596b07-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b92369f1-9c4d-4654-91c8-4c8925596b07-maas-api-service-ca\") pod \"b92369f1-9c4d-4654-91c8-4c8925596b07\" (UID: \"b92369f1-9c4d-4654-91c8-4c8925596b07\") " Oct 30 23:30:22.299474 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:22.299416 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b92369f1-9c4d-4654-91c8-4c8925596b07-kube-api-access-qpnt9\" (UniqueName: \"kubernetes.io/projected/b92369f1-9c4d-4654-91c8-4c8925596b07-kube-api-access-qpnt9\") pod \"b92369f1-9c4d-4654-91c8-4c8925596b07\" (UID: \"b92369f1-9c4d-4654-91c8-4c8925596b07\") " Oct 30 23:30:22.299541 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:22.299521 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36cdec6c-538d-4983-ab87-83a1ff62a222-maas-api-service-ca" pod "36cdec6c-538d-4983-ab87-83a1ff62a222" (UID: "36cdec6c-538d-4983-ab87-83a1ff62a222"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:30:22.300138 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:22.300111 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36cdec6c-538d-4983-ab87-83a1ff62a222-maas-api-service-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:30:22.300337 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:22.300310 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b92369f1-9c4d-4654-91c8-4c8925596b07-maas-api-service-ca" pod "b92369f1-9c4d-4654-91c8-4c8925596b07" (UID: "b92369f1-9c4d-4654-91c8-4c8925596b07"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:30:22.302003 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:22.301978 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36cdec6c-538d-4983-ab87-83a1ff62a222-kube-api-access-nwcns" pod "36cdec6c-538d-4983-ab87-83a1ff62a222" (UID: "36cdec6c-538d-4983-ab87-83a1ff62a222"). InnerVolumeSpecName "kube-api-access-nwcns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:30:22.302829 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:22.302807 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b92369f1-9c4d-4654-91c8-4c8925596b07-kube-api-access-qpnt9" pod "b92369f1-9c4d-4654-91c8-4c8925596b07" (UID: "b92369f1-9c4d-4654-91c8-4c8925596b07"). InnerVolumeSpecName "kube-api-access-qpnt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:30:22.403029 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:22.402943 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwcns\" (UniqueName: \"kubernetes.io/projected/36cdec6c-538d-4983-ab87-83a1ff62a222-kube-api-access-nwcns\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:30:22.403029 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:22.402986 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b92369f1-9c4d-4654-91c8-4c8925596b07-maas-api-service-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:30:22.403029 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:22.403005 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpnt9\" (UniqueName: \"kubernetes.io/projected/b92369f1-9c4d-4654-91c8-4c8925596b07-kube-api-access-qpnt9\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:30:22.505451 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:22.505408 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" Oct 30 23:30:22.506023 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:22.505984 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" Oct 30 23:30:22.508673 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:22.508648 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" Oct 30 23:30:22.877575 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:22.877545 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh" Oct 30 23:30:22.879301 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:22.879271 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh" event={"ID":"b92369f1-9c4d-4654-91c8-4c8925596b07","Type":"ContainerDied","Data":"89b9ff00dcd6941ea751965101db8cc93b9f9937c356a6697b649dfabf9c58a2"} Oct 30 23:30:22.879481 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:22.879368 4952 scope.go:122] "RemoveContainer" containerID="4c2a87dd66c954908d8d12bdf74e609df45c04b0b6f27996e747ba0b966f60ea" Oct 30 23:30:22.885924 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:22.885855 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548" Oct 30 23:30:22.886079 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:22.885957 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548" event={"ID":"36cdec6c-538d-4983-ab87-83a1ff62a222","Type":"ContainerDied","Data":"4cfe564e190786bb0e48c2644df8d7126b7182a96372d0d5b103c5bab21a9119"} Oct 30 23:30:22.925719 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:22.925664 4952 scope.go:122] "RemoveContainer" containerID="e6d91669d0abdf0adfc7992c4d1328e08629db5bc125d33dea90b0e7c6ba6c09" Oct 30 23:30:22.972681 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:22.972654 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84968a4b-45c2-4631-a9ae-2480b2ffbdda" path="/var/lib/kubelet/pods/84968a4b-45c2-4631-a9ae-2480b2ffbdda/volumes" Oct 30 23:30:23.048854 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:23.048826 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548"] Oct 30 23:30:23.053364 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:23.053341 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890050-gn548"] Oct 30 23:30:23.164498 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:23.164409 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh"] Oct 30 23:30:23.171026 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:23.171001 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890050-svflh"] Oct 30 23:30:24.826717 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:24.826692 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686" Oct 30 23:30:24.974197 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:24.974157 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36cdec6c-538d-4983-ab87-83a1ff62a222" path="/var/lib/kubelet/pods/36cdec6c-538d-4983-ab87-83a1ff62a222/volumes" Oct 30 23:30:24.974707 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:24.974684 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b92369f1-9c4d-4654-91c8-4c8925596b07" path="/var/lib/kubelet/pods/b92369f1-9c4d-4654-91c8-4c8925596b07/volumes" Oct 30 23:30:25.999275 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:25.999218 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:30:30.636605 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:30.636552 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:30:30.870051 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:30.870003 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:30:31.098691 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:31.098651 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:30:35.227165 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.227133 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz"] Oct 30 23:30:35.227756 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.227741 4952 cpu_manager.go:414] "containerMap: removing container" podUID="b92369f1-9c4d-4654-91c8-4c8925596b07" containerName="cleanup" Oct 30 23:30:35.227808 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.227757 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b92369f1-9c4d-4654-91c8-4c8925596b07" containerName="cleanup" Oct 30 23:30:35.227808 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.227781 4952 cpu_manager.go:414] "containerMap: removing container" podUID="b92369f1-9c4d-4654-91c8-4c8925596b07" containerName="cleanup" Oct 30 23:30:35.227808 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.227786 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b92369f1-9c4d-4654-91c8-4c8925596b07" containerName="cleanup" Oct 30 23:30:35.227808 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.227797 4952 cpu_manager.go:414] "containerMap: removing container" podUID="84968a4b-45c2-4631-a9ae-2480b2ffbdda" containerName="cleanup" Oct 30 23:30:35.227808 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.227802 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84968a4b-45c2-4631-a9ae-2480b2ffbdda" containerName="cleanup" Oct 30 23:30:35.227956 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.227812 4952 cpu_manager.go:414] "containerMap: removing container" podUID="36cdec6c-538d-4983-ab87-83a1ff62a222" containerName="cleanup" Oct 30 23:30:35.227956 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.227816 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36cdec6c-538d-4983-ab87-83a1ff62a222" containerName="cleanup" Oct 30 23:30:35.227956 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.227823 4952 cpu_manager.go:414] "containerMap: removing container" podUID="84968a4b-45c2-4631-a9ae-2480b2ffbdda" containerName="cleanup" Oct 30 23:30:35.227956 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.227828 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84968a4b-45c2-4631-a9ae-2480b2ffbdda" containerName="cleanup" Oct 30 23:30:35.227956 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.227842 4952 cpu_manager.go:414] "containerMap: removing container" podUID="36cdec6c-538d-4983-ab87-83a1ff62a222" containerName="cleanup" Oct 30 23:30:35.227956 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.227847 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36cdec6c-538d-4983-ab87-83a1ff62a222" containerName="cleanup" Oct 30 23:30:35.227956 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.227949 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="36cdec6c-538d-4983-ab87-83a1ff62a222" containerName="cleanup" Oct 30 23:30:35.227956 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.227957 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="36cdec6c-538d-4983-ab87-83a1ff62a222" containerName="cleanup" Oct 30 23:30:35.228212 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.227963 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="84968a4b-45c2-4631-a9ae-2480b2ffbdda" containerName="cleanup" Oct 30 23:30:35.228212 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.227970 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="b92369f1-9c4d-4654-91c8-4c8925596b07" containerName="cleanup" Oct 30 23:30:35.228212 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.227977 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="b92369f1-9c4d-4654-91c8-4c8925596b07" containerName="cleanup" Oct 30 23:30:35.228212 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.227984 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="36cdec6c-538d-4983-ab87-83a1ff62a222" containerName="cleanup" Oct 30 23:30:35.228212 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.227990 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="84968a4b-45c2-4631-a9ae-2480b2ffbdda" containerName="cleanup" Oct 30 23:30:35.233070 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.233031 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz" Oct 30 23:30:35.288230 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.288207 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz"] Oct 30 23:30:35.332586 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.332554 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2b6c82a0-serving-cert\"" Oct 30 23:30:35.378893 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.378864 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43ea043e-1ca0-497f-b8be-30e75effa2b0-metrics-certs\") pod \"maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz\" (UID: \"43ea043e-1ca0-497f-b8be-30e75effa2b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz" Oct 30 23:30:35.379003 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.378917 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l4cm\" (UniqueName: \"kubernetes.io/projected/43ea043e-1ca0-497f-b8be-30e75effa2b0-kube-api-access-4l4cm\") pod \"maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz\" (UID: \"43ea043e-1ca0-497f-b8be-30e75effa2b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz" Oct 30 23:30:35.379144 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.379125 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43ea043e-1ca0-497f-b8be-30e75effa2b0-maas-api-tls\") pod \"maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz\" (UID: \"43ea043e-1ca0-497f-b8be-30e75effa2b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz" Oct 30 23:30:35.379229 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.379207 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ea043e-1ca0-497f-b8be-30e75effa2b0-openshift-service-ca\") pod \"maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz\" (UID: \"43ea043e-1ca0-497f-b8be-30e75effa2b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz" Oct 30 23:30:35.482268 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.480513 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43ea043e-1ca0-497f-b8be-30e75effa2b0-metrics-certs\") pod \"maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz\" (UID: \"43ea043e-1ca0-497f-b8be-30e75effa2b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz" Oct 30 23:30:35.482268 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.480571 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l4cm\" (UniqueName: \"kubernetes.io/projected/43ea043e-1ca0-497f-b8be-30e75effa2b0-kube-api-access-4l4cm\") pod \"maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz\" (UID: \"43ea043e-1ca0-497f-b8be-30e75effa2b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz" Oct 30 23:30:35.482268 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.480731 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43ea043e-1ca0-497f-b8be-30e75effa2b0-maas-api-tls\") pod \"maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz\" (UID: \"43ea043e-1ca0-497f-b8be-30e75effa2b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz" Oct 30 23:30:35.482268 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.480759 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ea043e-1ca0-497f-b8be-30e75effa2b0-openshift-service-ca\") pod \"maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz\" (UID: \"43ea043e-1ca0-497f-b8be-30e75effa2b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz" Oct 30 23:30:35.482268 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.481781 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ea043e-1ca0-497f-b8be-30e75effa2b0-openshift-service-ca\") pod \"maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz\" (UID: \"43ea043e-1ca0-497f-b8be-30e75effa2b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz" Oct 30 23:30:35.482899 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:30:35.482875 4952 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-2b6c82a0-serving-cert: secret "maas-api-e2e-ait-2b6c82a0-serving-cert" not found Oct 30 23:30:35.482992 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:30:35.482956 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43ea043e-1ca0-497f-b8be-30e75effa2b0-maas-api-tls podName:43ea043e-1ca0-497f-b8be-30e75effa2b0 nodeName:}" failed. No retries permitted until 2026-10-30 23:30:35.982935958 +0000 UTC m=+1455.126766144 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/43ea043e-1ca0-497f-b8be-30e75effa2b0-maas-api-tls") pod "maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz" (UID: "43ea043e-1ca0-497f-b8be-30e75effa2b0") : secret "maas-api-e2e-ait-2b6c82a0-serving-cert" not found Oct 30 23:30:35.485156 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.485114 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43ea043e-1ca0-497f-b8be-30e75effa2b0-metrics-certs\") pod \"maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz\" (UID: \"43ea043e-1ca0-497f-b8be-30e75effa2b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz" Oct 30 23:30:35.507128 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.507096 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:30:35.540438 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.540412 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l4cm\" (UniqueName: \"kubernetes.io/projected/43ea043e-1ca0-497f-b8be-30e75effa2b0-kube-api-access-4l4cm\") pod \"maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz\" (UID: \"43ea043e-1ca0-497f-b8be-30e75effa2b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz" Oct 30 23:30:35.737572 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.737469 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:30:35.978072 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.978000 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:30:35.990065 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.989978 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43ea043e-1ca0-497f-b8be-30e75effa2b0-maas-api-tls\") pod \"maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz\" (UID: \"43ea043e-1ca0-497f-b8be-30e75effa2b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz" Oct 30 23:30:35.992780 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:35.992748 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43ea043e-1ca0-497f-b8be-30e75effa2b0-maas-api-tls\") pod \"maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz\" (UID: \"43ea043e-1ca0-497f-b8be-30e75effa2b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz" Oct 30 23:30:36.147101 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:36.147073 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz" Oct 30 23:30:36.533820 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:30:36.533793 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ea043e_1ca0_497f_b8be_30e75effa2b0.slice/crio-4374998c2322eee4111cf0c372bf283a2279d2c489dd48f85fe981c2e9862e5a WatchSource:0}: Error finding container 4374998c2322eee4111cf0c372bf283a2279d2c489dd48f85fe981c2e9862e5a: Status 404 returned error can't find the container with id 4374998c2322eee4111cf0c372bf283a2279d2c489dd48f85fe981c2e9862e5a Oct 30 23:30:36.534320 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:36.533964 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz"] Oct 30 23:30:37.004563 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:37.004526 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz" event={"ID":"43ea043e-1ca0-497f-b8be-30e75effa2b0","Type":"ContainerStarted","Data":"4374998c2322eee4111cf0c372bf283a2279d2c489dd48f85fe981c2e9862e5a"} Oct 30 23:30:38.013567 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:38.013532 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz" event={"ID":"43ea043e-1ca0-497f-b8be-30e75effa2b0","Type":"ContainerStarted","Data":"5695ad41fab7d99b90415aa0f124a5d743b55c7f4faa6bd71eaf6d83f0897d5d"} Oct 30 23:30:38.013943 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:38.013607 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz" Oct 30 23:30:38.130816 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:38.130750 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz" podStartSLOduration=2.315316783 podStartE2EDuration="3.130732644s" totalImagesPullingTime="815.415861ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:30:38.129515943 +0000 UTC m=+1457.273346168" watchObservedRunningTime="2026-10-30 23:30:38.130732644 +0000 UTC m=+1457.274562850" Oct 30 23:30:42.295582 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:42.295550 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:30:44.049804 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:44.049776 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz" Oct 30 23:30:45.033908 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:45.033880 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9"] Oct 30 23:30:45.034523 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:45.034509 4952 cpu_manager.go:414] "containerMap: removing container" podUID="b92369f1-9c4d-4654-91c8-4c8925596b07" containerName="cleanup" Oct 30 23:30:45.034575 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:45.034524 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b92369f1-9c4d-4654-91c8-4c8925596b07" containerName="cleanup" Oct 30 23:30:45.034575 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:45.034543 4952 cpu_manager.go:414] "containerMap: removing container" podUID="84968a4b-45c2-4631-a9ae-2480b2ffbdda" containerName="cleanup" Oct 30 23:30:45.034575 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:45.034548 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84968a4b-45c2-4631-a9ae-2480b2ffbdda" containerName="cleanup" Oct 30 23:30:45.034575 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:45.034564 4952 cpu_manager.go:414] "containerMap: removing container" podUID="36cdec6c-538d-4983-ab87-83a1ff62a222" containerName="cleanup" Oct 30 23:30:45.034575 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:45.034569 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36cdec6c-538d-4983-ab87-83a1ff62a222" containerName="cleanup" Oct 30 23:30:45.034733 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:45.034671 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="84968a4b-45c2-4631-a9ae-2480b2ffbdda" containerName="cleanup" Oct 30 23:30:45.034733 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:45.034684 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="b92369f1-9c4d-4654-91c8-4c8925596b07" containerName="cleanup" Oct 30 23:30:45.038428 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:45.038413 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9" Oct 30 23:30:45.090541 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:45.090509 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-5f9ec6be-serving-cert\"" Oct 30 23:30:45.096361 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:45.096336 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4212bd64-9888-4ed7-815a-0a502bfdfb6c-maas-api-tls\") pod \"maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9\" (UID: \"4212bd64-9888-4ed7-815a-0a502bfdfb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9" Oct 30 23:30:45.096549 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:45.096532 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx85p\" (UniqueName: \"kubernetes.io/projected/4212bd64-9888-4ed7-815a-0a502bfdfb6c-kube-api-access-vx85p\") pod \"maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9\" (UID: \"4212bd64-9888-4ed7-815a-0a502bfdfb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9" Oct 30 23:30:45.096604 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:45.096585 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4212bd64-9888-4ed7-815a-0a502bfdfb6c-openshift-service-ca\") pod \"maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9\" (UID: \"4212bd64-9888-4ed7-815a-0a502bfdfb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9" Oct 30 23:30:45.096711 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:45.096699 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4212bd64-9888-4ed7-815a-0a502bfdfb6c-metrics-certs\") pod \"maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9\" (UID: \"4212bd64-9888-4ed7-815a-0a502bfdfb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9" Oct 30 23:30:45.144671 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:45.144638 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9"] Oct 30 23:30:45.197818 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:45.197788 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx85p\" (UniqueName: \"kubernetes.io/projected/4212bd64-9888-4ed7-815a-0a502bfdfb6c-kube-api-access-vx85p\") pod \"maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9\" (UID: \"4212bd64-9888-4ed7-815a-0a502bfdfb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9" Oct 30 23:30:45.197970 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:45.197849 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4212bd64-9888-4ed7-815a-0a502bfdfb6c-openshift-service-ca\") pod \"maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9\" (UID: \"4212bd64-9888-4ed7-815a-0a502bfdfb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9" Oct 30 23:30:45.197970 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:45.197939 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4212bd64-9888-4ed7-815a-0a502bfdfb6c-metrics-certs\") pod \"maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9\" (UID: \"4212bd64-9888-4ed7-815a-0a502bfdfb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9" Oct 30 23:30:45.198112 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:45.197978 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4212bd64-9888-4ed7-815a-0a502bfdfb6c-maas-api-tls\") pod \"maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9\" (UID: \"4212bd64-9888-4ed7-815a-0a502bfdfb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9" Oct 30 23:30:45.198177 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:30:45.198133 4952 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-5f9ec6be-serving-cert: secret "maas-api-e2e-mt-5f9ec6be-serving-cert" not found Oct 30 23:30:45.198217 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:30:45.198188 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4212bd64-9888-4ed7-815a-0a502bfdfb6c-maas-api-tls podName:4212bd64-9888-4ed7-815a-0a502bfdfb6c nodeName:}" failed. No retries permitted until 2026-10-30 23:30:45.698174571 +0000 UTC m=+1464.842004755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4212bd64-9888-4ed7-815a-0a502bfdfb6c-maas-api-tls") pod "maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9" (UID: "4212bd64-9888-4ed7-815a-0a502bfdfb6c") : secret "maas-api-e2e-mt-5f9ec6be-serving-cert" not found Oct 30 23:30:45.198572 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:45.198545 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4212bd64-9888-4ed7-815a-0a502bfdfb6c-openshift-service-ca\") pod \"maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9\" (UID: \"4212bd64-9888-4ed7-815a-0a502bfdfb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9" Oct 30 23:30:45.200571 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:45.200550 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4212bd64-9888-4ed7-815a-0a502bfdfb6c-metrics-certs\") pod \"maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9\" (UID: \"4212bd64-9888-4ed7-815a-0a502bfdfb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9" Oct 30 23:30:45.255203 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:45.255176 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx85p\" (UniqueName: \"kubernetes.io/projected/4212bd64-9888-4ed7-815a-0a502bfdfb6c-kube-api-access-vx85p\") pod \"maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9\" (UID: \"4212bd64-9888-4ed7-815a-0a502bfdfb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9" Oct 30 23:30:45.319528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:45.319407 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:30:45.554330 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:45.554292 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:30:45.706111 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:45.706075 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4212bd64-9888-4ed7-815a-0a502bfdfb6c-maas-api-tls\") pod \"maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9\" (UID: \"4212bd64-9888-4ed7-815a-0a502bfdfb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9" Oct 30 23:30:45.708696 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:45.708673 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4212bd64-9888-4ed7-815a-0a502bfdfb6c-maas-api-tls\") pod \"maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9\" (UID: \"4212bd64-9888-4ed7-815a-0a502bfdfb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9" Oct 30 23:30:45.780510 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:45.780487 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:30:45.952512 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:45.952476 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9" Oct 30 23:30:46.038411 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:46.038369 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:30:46.330182 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:30:46.330155 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4212bd64_9888_4ed7_815a_0a502bfdfb6c.slice/crio-b4707d8206f8717c2386a28e937c8979a1a9935ae3914c60c5a69378336aea33 WatchSource:0}: Error finding container b4707d8206f8717c2386a28e937c8979a1a9935ae3914c60c5a69378336aea33: Status 404 returned error can't find the container with id b4707d8206f8717c2386a28e937c8979a1a9935ae3914c60c5a69378336aea33 Oct 30 23:30:46.330691 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:46.330642 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9"] Oct 30 23:30:47.121462 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:47.121423 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9" event={"ID":"4212bd64-9888-4ed7-815a-0a502bfdfb6c","Type":"ContainerStarted","Data":"b4707d8206f8717c2386a28e937c8979a1a9935ae3914c60c5a69378336aea33"} Oct 30 23:30:48.146066 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:48.141601 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9" event={"ID":"4212bd64-9888-4ed7-815a-0a502bfdfb6c","Type":"ContainerStarted","Data":"7a1235260751dd38f5124172dffc890c48058b7298153acf17638baaa8918f90"} Oct 30 23:30:48.146066 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:48.142265 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9" Oct 30 23:30:48.258895 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:48.258840 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9" podStartSLOduration=3.371906473 podStartE2EDuration="4.258825763s" totalImagesPullingTime="886.91929ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:30:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:30:48.256695865 +0000 UTC m=+1467.400526071" watchObservedRunningTime="2026-10-30 23:30:48.258825763 +0000 UTC m=+1467.402655968" Oct 30 23:30:50.298161 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:50.298114 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:30:51.142983 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:30:51.142948 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:30:51.819295 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:51.819243 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:30:54.155814 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:54.155788 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9" Oct 30 23:30:55.191923 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:55.191892 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:30:57.375560 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:57.375517 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:30:59.975363 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:30:59.975273 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:31:01.845207 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:01.845173 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:31:02.601086 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:02.601029 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:31:04.204063 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:04.201655 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:31:04.769688 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:04.769643 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:31:05.015026 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:05.014994 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:31:05.262315 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:05.262278 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:31:09.779238 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:09.779208 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:31:13.251225 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:13.251185 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:31:14.694807 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:14.694768 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:31:19.003560 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:19.003523 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:31:19.679842 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:19.679811 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:31:21.144716 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:31:21.144681 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:31:21.179861 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:21.179820 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft_91651ce2-4673-4ced-b26d-e94ff8829d11/extract/0.log" Oct 30 23:31:21.188576 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:21.188550 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft_91651ce2-4673-4ced-b26d-e94ff8829d11/extract/0.log" Oct 30 23:31:21.190741 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:21.190701 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/2.log" Oct 30 23:31:21.196919 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:21.196901 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 30 23:31:21.198791 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:21.198768 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/2.log" Oct 30 23:31:21.204707 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:21.204690 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 30 23:31:24.601104 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:24.601070 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:31:26.447000 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:26.446956 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:31:29.849420 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:29.849376 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:31:35.231557 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:35.231515 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:31:40.467351 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:40.467316 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:31:42.591866 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:42.591830 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:31:42.834802 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:42.834770 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:31:45.792472 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:45.792425 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:31:50.974862 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:50.974812 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:31:51.144820 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:31:51.144781 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:31:55.316801 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:55.316717 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:31:55.549952 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:55.549919 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:31:56.179854 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:56.179817 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:31:57.684353 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:57.684319 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:31:58.538802 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:58.538762 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:31:58.540574 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:58.540550 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8"] Oct 30 23:31:58.544949 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:58.544930 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" Oct 30 23:31:58.604760 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:58.604728 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323943ac-0c8a-4612-84d0-3e009c8436ab-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rnmq8\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" Oct 30 23:31:58.604897 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:58.604773 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323943ac-0c8a-4612-84d0-3e009c8436ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rnmq8\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" Oct 30 23:31:58.604897 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:58.604871 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323943ac-0c8a-4612-84d0-3e009c8436ab-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rnmq8\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" Oct 30 23:31:58.605009 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:58.604909 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323943ac-0c8a-4612-84d0-3e009c8436ab-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rnmq8\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" Oct 30 23:31:58.605092 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:58.605007 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323943ac-0c8a-4612-84d0-3e009c8436ab-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rnmq8\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" Oct 30 23:31:58.605226 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:58.605203 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2zv9\" (UniqueName: \"kubernetes.io/projected/323943ac-0c8a-4612-84d0-3e009c8436ab-kube-api-access-h2zv9\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rnmq8\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" Oct 30 23:31:58.605328 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:58.605252 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323943ac-0c8a-4612-84d0-3e009c8436ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rnmq8\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" Oct 30 23:31:58.605405 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:58.605385 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323943ac-0c8a-4612-84d0-3e009c8436ab-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rnmq8\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" Oct 30 23:31:58.605458 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:58.605427 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323943ac-0c8a-4612-84d0-3e009c8436ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rnmq8\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" Oct 30 23:31:58.706873 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:58.706844 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323943ac-0c8a-4612-84d0-3e009c8436ab-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rnmq8\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" Oct 30 23:31:58.707405 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:58.706884 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323943ac-0c8a-4612-84d0-3e009c8436ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rnmq8\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" Oct 30 23:31:58.707405 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:58.706914 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323943ac-0c8a-4612-84d0-3e009c8436ab-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rnmq8\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" Oct 30 23:31:58.707405 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:58.706941 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323943ac-0c8a-4612-84d0-3e009c8436ab-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rnmq8\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" Oct 30 23:31:58.707405 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:58.706987 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323943ac-0c8a-4612-84d0-3e009c8436ab-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rnmq8\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" Oct 30 23:31:58.707405 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:58.707064 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2zv9\" (UniqueName: \"kubernetes.io/projected/323943ac-0c8a-4612-84d0-3e009c8436ab-kube-api-access-h2zv9\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rnmq8\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" Oct 30 23:31:58.707405 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:58.707089 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323943ac-0c8a-4612-84d0-3e009c8436ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rnmq8\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" Oct 30 23:31:58.707405 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:58.707148 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323943ac-0c8a-4612-84d0-3e009c8436ab-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rnmq8\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" Oct 30 23:31:58.707405 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:58.707171 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323943ac-0c8a-4612-84d0-3e009c8436ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rnmq8\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" Oct 30 23:31:58.707405 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:58.707251 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323943ac-0c8a-4612-84d0-3e009c8436ab-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rnmq8\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" Oct 30 23:31:58.707908 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:58.707599 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323943ac-0c8a-4612-84d0-3e009c8436ab-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rnmq8\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" Oct 30 23:31:58.707908 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:58.707785 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323943ac-0c8a-4612-84d0-3e009c8436ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rnmq8\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" Oct 30 23:31:58.708215 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:58.708197 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323943ac-0c8a-4612-84d0-3e009c8436ab-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rnmq8\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" Oct 30 23:31:58.708321 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:58.708269 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323943ac-0c8a-4612-84d0-3e009c8436ab-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rnmq8\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" Oct 30 23:31:58.710432 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:58.710409 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323943ac-0c8a-4612-84d0-3e009c8436ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rnmq8\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" Oct 30 23:31:58.710648 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:58.710630 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323943ac-0c8a-4612-84d0-3e009c8436ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rnmq8\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" Oct 30 23:31:58.758225 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:58.758195 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8"] Oct 30 23:31:58.771442 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:58.771412 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323943ac-0c8a-4612-84d0-3e009c8436ab-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rnmq8\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" Oct 30 23:31:58.771795 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:58.771773 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2zv9\" (UniqueName: \"kubernetes.io/projected/323943ac-0c8a-4612-84d0-3e009c8436ab-kube-api-access-h2zv9\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rnmq8\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" Oct 30 23:31:58.778871 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:58.778840 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:31:58.859656 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:58.859631 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" Oct 30 23:31:59.146691 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:59.146611 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:31:59.293817 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:31:59.293790 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod323943ac_0c8a_4612_84d0_3e009c8436ab.slice/crio-d4cf034f8a1036fd52f8133b940b9bc8f7ad060775f365fb05a27621ff17f28f WatchSource:0}: Error finding container d4cf034f8a1036fd52f8133b940b9bc8f7ad060775f365fb05a27621ff17f28f: Status 404 returned error can't find the container with id d4cf034f8a1036fd52f8133b940b9bc8f7ad060775f365fb05a27621ff17f28f Oct 30 23:31:59.294096 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:59.294075 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8"] Oct 30 23:31:59.296249 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:59.296221 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:31:59.296322 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:59.296282 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:31:59.296322 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:59.296312 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:31:59.812563 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:59.812530 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" event={"ID":"323943ac-0c8a-4612-84d0-3e009c8436ab","Type":"ContainerStarted","Data":"88238ded6d97e7f08a0a80cf8395c3f048946128803a307141b82a7844d8398b"} Oct 30 23:31:59.813094 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:59.812582 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" event={"ID":"323943ac-0c8a-4612-84d0-3e009c8436ab","Type":"ContainerStarted","Data":"d4cf034f8a1036fd52f8133b940b9bc8f7ad060775f365fb05a27621ff17f28f"} Oct 30 23:31:59.859733 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:59.859703 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" Oct 30 23:31:59.985350 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:31:59.985302 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" podStartSLOduration=1.9852878139999999 podStartE2EDuration="1.985287814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:31:59.981987898 +0000 UTC m=+1539.125818117" watchObservedRunningTime="2026-10-30 23:31:59.985287814 +0000 UTC m=+1539.129118019" Oct 30 23:32:00.861538 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:00.861504 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-rnmq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 23:32:00.861960 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:00.861555 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" podUID="323943ac-0c8a-4612-84d0-3e009c8436ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 23:32:01.509064 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:01.506882 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:01.860596 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:01.860569 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-rnmq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 23:32:01.860754 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:01.860618 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" podUID="323943ac-0c8a-4612-84d0-3e009c8436ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 23:32:02.635058 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:02.617298 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:02.843086 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:02.843000 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:02.860646 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:02.860613 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-rnmq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 23:32:02.860840 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:02.860677 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" podUID="323943ac-0c8a-4612-84d0-3e009c8436ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 23:32:03.864059 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:03.861016 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-rnmq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 23:32:03.864059 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:03.861246 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" podUID="323943ac-0c8a-4612-84d0-3e009c8436ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 23:32:04.859857 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:04.859826 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-rnmq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 23:32:04.860024 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:04.859873 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" podUID="323943ac-0c8a-4612-84d0-3e009c8436ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 23:32:05.331331 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:05.331239 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:05.859878 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:05.859846 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-rnmq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 23:32:05.860079 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:05.859903 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" podUID="323943ac-0c8a-4612-84d0-3e009c8436ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 23:32:06.178804 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:06.178717 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:32:06.490135 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:06.490008 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:06.860289 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:06.860252 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-rnmq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 23:32:06.860458 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:06.860301 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" podUID="323943ac-0c8a-4612-84d0-3e009c8436ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 23:32:08.234367 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:08.234336 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" Oct 30 23:32:08.234869 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:08.234831 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" Oct 30 23:32:08.235886 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:08.235865 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" Oct 30 23:32:11.299529 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:11.299495 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:16.177191 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:16.177155 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:16.414670 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:16.414639 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:17.734416 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:17.734383 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:18.035132 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:18.035023 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:19.338478 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:19.338441 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:21.144058 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:32:21.144011 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:32:21.681369 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:21.681332 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:25.727096 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:25.727065 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686"] Oct 30 23:32:25.727579 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:25.727259 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686" podUID="4863fa66-8c17-49c8-be19-01693e905cd3" containerName="maas-api" containerID="cri-o://eeb0dae4ce8033e253176b7f39eddc47cf4672548d80a20d7e9385a547bb0340" gracePeriod=30 Oct 30 23:32:25.933632 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:25.933583 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:26.048257 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:26.048172 4952 generic.go:356] "Generic (PLEG): container finished" podID="4863fa66-8c17-49c8-be19-01693e905cd3" containerID="eeb0dae4ce8033e253176b7f39eddc47cf4672548d80a20d7e9385a547bb0340" exitCode=0 Oct 30 23:32:26.048558 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:26.048313 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686" event={"ID":"4863fa66-8c17-49c8-be19-01693e905cd3","Type":"ContainerDied","Data":"eeb0dae4ce8033e253176b7f39eddc47cf4672548d80a20d7e9385a547bb0340"} Oct 30 23:32:26.161577 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:26.161540 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686" Oct 30 23:32:26.297934 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:26.297902 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4863fa66-8c17-49c8-be19-01693e905cd3-kube-api-access-jl7ww\" (UniqueName: \"kubernetes.io/projected/4863fa66-8c17-49c8-be19-01693e905cd3-kube-api-access-jl7ww\") pod \"4863fa66-8c17-49c8-be19-01693e905cd3\" (UID: \"4863fa66-8c17-49c8-be19-01693e905cd3\") " Oct 30 23:32:26.298229 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:26.298207 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4863fa66-8c17-49c8-be19-01693e905cd3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4863fa66-8c17-49c8-be19-01693e905cd3-metrics-certs\") pod \"4863fa66-8c17-49c8-be19-01693e905cd3\" (UID: \"4863fa66-8c17-49c8-be19-01693e905cd3\") " Oct 30 23:32:26.298440 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:26.298394 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4863fa66-8c17-49c8-be19-01693e905cd3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4863fa66-8c17-49c8-be19-01693e905cd3-openshift-service-ca\") pod \"4863fa66-8c17-49c8-be19-01693e905cd3\" (UID: \"4863fa66-8c17-49c8-be19-01693e905cd3\") " Oct 30 23:32:26.298660 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:26.298642 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4863fa66-8c17-49c8-be19-01693e905cd3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4863fa66-8c17-49c8-be19-01693e905cd3-maas-api-tls\") pod \"4863fa66-8c17-49c8-be19-01693e905cd3\" (UID: \"4863fa66-8c17-49c8-be19-01693e905cd3\") " Oct 30 23:32:26.300008 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:26.299985 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4863fa66-8c17-49c8-be19-01693e905cd3-openshift-service-ca" pod "4863fa66-8c17-49c8-be19-01693e905cd3" (UID: "4863fa66-8c17-49c8-be19-01693e905cd3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:32:26.301008 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:26.300982 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4863fa66-8c17-49c8-be19-01693e905cd3-kube-api-access-jl7ww" pod "4863fa66-8c17-49c8-be19-01693e905cd3" (UID: "4863fa66-8c17-49c8-be19-01693e905cd3"). InnerVolumeSpecName "kube-api-access-jl7ww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:32:26.303771 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:26.303745 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4863fa66-8c17-49c8-be19-01693e905cd3-maas-api-tls" pod "4863fa66-8c17-49c8-be19-01693e905cd3" (UID: "4863fa66-8c17-49c8-be19-01693e905cd3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:32:26.303980 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:26.303960 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4863fa66-8c17-49c8-be19-01693e905cd3-metrics-certs" pod "4863fa66-8c17-49c8-be19-01693e905cd3" (UID: "4863fa66-8c17-49c8-be19-01693e905cd3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:32:26.401199 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:26.401162 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl7ww\" (UniqueName: \"kubernetes.io/projected/4863fa66-8c17-49c8-be19-01693e905cd3-kube-api-access-jl7ww\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:32:26.401199 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:26.401198 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4863fa66-8c17-49c8-be19-01693e905cd3-metrics-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:32:26.401403 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:26.401211 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4863fa66-8c17-49c8-be19-01693e905cd3-openshift-service-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:32:26.401403 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:26.401227 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4863fa66-8c17-49c8-be19-01693e905cd3-maas-api-tls\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:32:27.058635 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:27.058580 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686" Oct 30 23:32:27.059299 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:27.058724 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686" event={"ID":"4863fa66-8c17-49c8-be19-01693e905cd3","Type":"ContainerDied","Data":"f84e37e9b1b57b8155ca13f1b8cb741937241837ac900fe712bd472999f8e28e"} Oct 30 23:32:27.059299 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:27.058782 4952 scope.go:122] "RemoveContainer" containerID="eeb0dae4ce8033e253176b7f39eddc47cf4672548d80a20d7e9385a547bb0340" Oct 30 23:32:27.256165 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:27.255897 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686"] Oct 30 23:32:27.285531 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:27.285498 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49d07c83-6c46bd9d5c-zt686"] Oct 30 23:32:27.525883 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:27.525788 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:27.761555 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:27.761522 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:28.009325 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:28.009293 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:28.258126 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:28.255690 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:28.534284 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:28.534248 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:28.967222 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:28.967185 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:28.973331 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:28.973308 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4863fa66-8c17-49c8-be19-01693e905cd3" path="/var/lib/kubelet/pods/4863fa66-8c17-49c8-be19-01693e905cd3/volumes" Oct 30 23:32:29.277149 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:29.277021 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:29.972436 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:29.968155 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:30.300851 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:30.300755 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:30.645303 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:30.645270 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:30.934200 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:30.934116 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:33.186496 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:33.186460 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:34.415657 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:34.415618 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:34.647642 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:34.647609 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:37.068626 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:37.068595 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:37.294434 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:37.294391 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:37.825338 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:37.825303 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:38.895825 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:38.895790 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz"] Oct 30 23:32:38.896455 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:38.896088 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz" podUID="43ea043e-1ca0-497f-b8be-30e75effa2b0" containerName="maas-api" containerID="cri-o://5695ad41fab7d99b90415aa0f124a5d743b55c7f4faa6bd71eaf6d83f0897d5d" gracePeriod=30 Oct 30 23:32:39.045209 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:39.045174 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz" podUID="43ea043e-1ca0-497f-b8be-30e75effa2b0" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.108:8443/health\": dial tcp 10.132.0.108:8443: connect: connection refused" Oct 30 23:32:39.126731 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:39.126702 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:39.227550 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:39.227191 4952 generic.go:356] "Generic (PLEG): container finished" podID="43ea043e-1ca0-497f-b8be-30e75effa2b0" containerID="5695ad41fab7d99b90415aa0f124a5d743b55c7f4faa6bd71eaf6d83f0897d5d" exitCode=0 Oct 30 23:32:39.227550 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:39.227331 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz" event={"ID":"43ea043e-1ca0-497f-b8be-30e75effa2b0","Type":"ContainerDied","Data":"5695ad41fab7d99b90415aa0f124a5d743b55c7f4faa6bd71eaf6d83f0897d5d"} Oct 30 23:32:39.295632 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:39.295606 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz" Oct 30 23:32:39.347985 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:39.347954 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43ea043e-1ca0-497f-b8be-30e75effa2b0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ea043e-1ca0-497f-b8be-30e75effa2b0-openshift-service-ca\") pod \"43ea043e-1ca0-497f-b8be-30e75effa2b0\" (UID: \"43ea043e-1ca0-497f-b8be-30e75effa2b0\") " Oct 30 23:32:39.348201 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:39.348018 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43ea043e-1ca0-497f-b8be-30e75effa2b0-kube-api-access-4l4cm\" (UniqueName: \"kubernetes.io/projected/43ea043e-1ca0-497f-b8be-30e75effa2b0-kube-api-access-4l4cm\") pod \"43ea043e-1ca0-497f-b8be-30e75effa2b0\" (UID: \"43ea043e-1ca0-497f-b8be-30e75effa2b0\") " Oct 30 23:32:39.348201 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:39.348069 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43ea043e-1ca0-497f-b8be-30e75effa2b0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43ea043e-1ca0-497f-b8be-30e75effa2b0-maas-api-tls\") pod \"43ea043e-1ca0-497f-b8be-30e75effa2b0\" (UID: \"43ea043e-1ca0-497f-b8be-30e75effa2b0\") " Oct 30 23:32:39.348201 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:39.348103 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43ea043e-1ca0-497f-b8be-30e75effa2b0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/43ea043e-1ca0-497f-b8be-30e75effa2b0-metrics-certs\") pod \"43ea043e-1ca0-497f-b8be-30e75effa2b0\" (UID: \"43ea043e-1ca0-497f-b8be-30e75effa2b0\") " Oct 30 23:32:39.351296 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:39.351262 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ea043e-1ca0-497f-b8be-30e75effa2b0-kube-api-access-4l4cm" pod "43ea043e-1ca0-497f-b8be-30e75effa2b0" (UID: "43ea043e-1ca0-497f-b8be-30e75effa2b0"). InnerVolumeSpecName "kube-api-access-4l4cm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:32:39.351455 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:39.351421 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ea043e-1ca0-497f-b8be-30e75effa2b0-openshift-service-ca" pod "43ea043e-1ca0-497f-b8be-30e75effa2b0" (UID: "43ea043e-1ca0-497f-b8be-30e75effa2b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:32:39.351948 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:39.351923 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43ea043e-1ca0-497f-b8be-30e75effa2b0-metrics-certs" pod "43ea043e-1ca0-497f-b8be-30e75effa2b0" (UID: "43ea043e-1ca0-497f-b8be-30e75effa2b0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:32:39.352840 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:39.352815 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43ea043e-1ca0-497f-b8be-30e75effa2b0-maas-api-tls" pod "43ea043e-1ca0-497f-b8be-30e75effa2b0" (UID: "43ea043e-1ca0-497f-b8be-30e75effa2b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:32:39.463771 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:39.462866 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:39.465821 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:39.465762 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ea043e-1ca0-497f-b8be-30e75effa2b0-openshift-service-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:32:39.465998 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:39.465983 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l4cm\" (UniqueName: \"kubernetes.io/projected/43ea043e-1ca0-497f-b8be-30e75effa2b0-kube-api-access-4l4cm\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:32:39.466165 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:39.466150 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43ea043e-1ca0-497f-b8be-30e75effa2b0-maas-api-tls\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:32:39.466270 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:39.466258 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43ea043e-1ca0-497f-b8be-30e75effa2b0-metrics-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:32:39.777450 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:39.777414 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:40.150297 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:40.150263 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:40.241581 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:40.241549 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz" event={"ID":"43ea043e-1ca0-497f-b8be-30e75effa2b0","Type":"ContainerDied","Data":"4374998c2322eee4111cf0c372bf283a2279d2c489dd48f85fe981c2e9862e5a"} Oct 30 23:32:40.241784 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:40.241625 4952 scope.go:122] "RemoveContainer" containerID="5695ad41fab7d99b90415aa0f124a5d743b55c7f4faa6bd71eaf6d83f0897d5d" Oct 30 23:32:40.241784 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:40.241641 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz" Oct 30 23:32:40.415772 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:40.415679 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz"] Oct 30 23:32:40.466810 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:40.466769 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2b6c82a0-6cbcfb9997-jhxmz"] Oct 30 23:32:40.850106 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:40.850068 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:40.973682 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:40.973655 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43ea043e-1ca0-497f-b8be-30e75effa2b0" path="/var/lib/kubelet/pods/43ea043e-1ca0-497f-b8be-30e75effa2b0/volumes" Oct 30 23:32:41.166860 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:41.166782 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:41.405824 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:41.405784 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:41.631469 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:41.631440 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:42.122902 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:42.122844 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:42.868147 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:42.865872 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:43.093527 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:43.090492 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:43.869979 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:43.869942 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:44.104126 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:44.104093 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:46.723289 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:46.723257 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:47.414643 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:47.414599 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:47.653983 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:47.653937 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:50.703591 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:50.703541 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:50.937507 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:50.937476 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:51.169844 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:32:51.169789 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:32:52.124316 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:52.124281 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:53.449125 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:53.449088 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:32:54.894326 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:54.894289 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:32:57.304773 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:32:57.304741 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:00.379705 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:00.379672 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:00.954077 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:00.954027 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:01.176030 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:01.175998 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:02.191548 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:02.191512 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:06.168518 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:06.168472 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:06.463474 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:06.463384 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:08.181840 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:08.181807 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:08.527976 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:08.527873 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:08.806478 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:08.806356 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:09.677335 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:09.677294 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:10.526391 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:10.526351 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:10.792055 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:10.791948 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:11.028291 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:11.028253 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:11.349541 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:11.349506 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:13.423361 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:13.423294 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:15.152009 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:15.151971 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm"] Oct 30 23:33:15.152532 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:15.152201 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" podUID="5889f46b-2aa6-4eab-a3e4-171b81b804de" containerName="istio-proxy" containerID="cri-o://a96c29f9b629337402ea621829c72e5a31b1e7ab2eb79ab1f496871cb288fc85" gracePeriod=30 Oct 30 23:33:17.131550 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:17.131517 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:17.375772 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:17.375744 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:17.679704 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:17.679662 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:18.378240 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:18.378209 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:19.690712 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:19.690681 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:20.049464 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.049379 4952 patch_prober.go:29] interesting pod/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 23:33:20.049464 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.049441 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" podUID="5889f46b-2aa6-4eab-a3e4-171b81b804de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:33:20.463705 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.463681 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" Oct 30 23:33:20.584782 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.584750 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5889f46b-2aa6-4eab-a3e4-171b81b804de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5889f46b-2aa6-4eab-a3e4-171b81b804de-istiod-ca-cert\") pod \"5889f46b-2aa6-4eab-a3e4-171b81b804de\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " Oct 30 23:33:20.584934 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.584854 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5889f46b-2aa6-4eab-a3e4-171b81b804de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5889f46b-2aa6-4eab-a3e4-171b81b804de-istio-podinfo\") pod \"5889f46b-2aa6-4eab-a3e4-171b81b804de\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " Oct 30 23:33:20.584934 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.584899 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5889f46b-2aa6-4eab-a3e4-171b81b804de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5889f46b-2aa6-4eab-a3e4-171b81b804de-istio-envoy\") pod \"5889f46b-2aa6-4eab-a3e4-171b81b804de\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " Oct 30 23:33:20.585085 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.584993 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5889f46b-2aa6-4eab-a3e4-171b81b804de-kube-api-access-82d84\" (UniqueName: \"kubernetes.io/projected/5889f46b-2aa6-4eab-a3e4-171b81b804de-kube-api-access-82d84\") pod \"5889f46b-2aa6-4eab-a3e4-171b81b804de\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " Oct 30 23:33:20.585085 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.585079 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5889f46b-2aa6-4eab-a3e4-171b81b804de-istio-token\" (UniqueName: \"kubernetes.io/projected/5889f46b-2aa6-4eab-a3e4-171b81b804de-istio-token\") pod \"5889f46b-2aa6-4eab-a3e4-171b81b804de\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " Oct 30 23:33:20.585233 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.585131 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5889f46b-2aa6-4eab-a3e4-171b81b804de-istiod-ca-cert" pod "5889f46b-2aa6-4eab-a3e4-171b81b804de" (UID: "5889f46b-2aa6-4eab-a3e4-171b81b804de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:33:20.585233 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.585152 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5889f46b-2aa6-4eab-a3e4-171b81b804de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5889f46b-2aa6-4eab-a3e4-171b81b804de-credential-socket\") pod \"5889f46b-2aa6-4eab-a3e4-171b81b804de\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " Oct 30 23:33:20.585335 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.585280 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5889f46b-2aa6-4eab-a3e4-171b81b804de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5889f46b-2aa6-4eab-a3e4-171b81b804de-istio-data\") pod \"5889f46b-2aa6-4eab-a3e4-171b81b804de\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " Oct 30 23:33:20.585389 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.585330 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5889f46b-2aa6-4eab-a3e4-171b81b804de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5889f46b-2aa6-4eab-a3e4-171b81b804de-workload-socket\") pod \"5889f46b-2aa6-4eab-a3e4-171b81b804de\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " Oct 30 23:33:20.585442 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.585390 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5889f46b-2aa6-4eab-a3e4-171b81b804de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5889f46b-2aa6-4eab-a3e4-171b81b804de-workload-certs\") pod \"5889f46b-2aa6-4eab-a3e4-171b81b804de\" (UID: \"5889f46b-2aa6-4eab-a3e4-171b81b804de\") " Oct 30 23:33:20.585536 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.585487 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5889f46b-2aa6-4eab-a3e4-171b81b804de-credential-socket" pod "5889f46b-2aa6-4eab-a3e4-171b81b804de" (UID: "5889f46b-2aa6-4eab-a3e4-171b81b804de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:33:20.585536 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.585512 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5889f46b-2aa6-4eab-a3e4-171b81b804de-istio-data" pod "5889f46b-2aa6-4eab-a3e4-171b81b804de" (UID: "5889f46b-2aa6-4eab-a3e4-171b81b804de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:33:20.585936 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.585734 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5889f46b-2aa6-4eab-a3e4-171b81b804de-workload-socket" pod "5889f46b-2aa6-4eab-a3e4-171b81b804de" (UID: "5889f46b-2aa6-4eab-a3e4-171b81b804de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:33:20.586006 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.585967 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5889f46b-2aa6-4eab-a3e4-171b81b804de-credential-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:33:20.586006 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.585981 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5889f46b-2aa6-4eab-a3e4-171b81b804de-istio-data\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:33:20.586006 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.585990 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5889f46b-2aa6-4eab-a3e4-171b81b804de-workload-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:33:20.586006 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.585987 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5889f46b-2aa6-4eab-a3e4-171b81b804de-workload-certs" pod "5889f46b-2aa6-4eab-a3e4-171b81b804de" (UID: "5889f46b-2aa6-4eab-a3e4-171b81b804de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:33:20.586006 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.585999 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5889f46b-2aa6-4eab-a3e4-171b81b804de-istiod-ca-cert\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:33:20.587425 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.587402 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5889f46b-2aa6-4eab-a3e4-171b81b804de-istio-envoy" pod "5889f46b-2aa6-4eab-a3e4-171b81b804de" (UID: "5889f46b-2aa6-4eab-a3e4-171b81b804de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:33:20.587528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.587504 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5889f46b-2aa6-4eab-a3e4-171b81b804de-istio-token" pod "5889f46b-2aa6-4eab-a3e4-171b81b804de" (UID: "5889f46b-2aa6-4eab-a3e4-171b81b804de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:33:20.587787 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.587770 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5889f46b-2aa6-4eab-a3e4-171b81b804de-istio-podinfo" pod "5889f46b-2aa6-4eab-a3e4-171b81b804de" (UID: "5889f46b-2aa6-4eab-a3e4-171b81b804de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:33:20.587836 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.587792 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5889f46b-2aa6-4eab-a3e4-171b81b804de-kube-api-access-82d84" pod "5889f46b-2aa6-4eab-a3e4-171b81b804de" (UID: "5889f46b-2aa6-4eab-a3e4-171b81b804de"). InnerVolumeSpecName "kube-api-access-82d84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:33:20.687306 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.687239 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5889f46b-2aa6-4eab-a3e4-171b81b804de-workload-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:33:20.687306 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.687264 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5889f46b-2aa6-4eab-a3e4-171b81b804de-istio-podinfo\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:33:20.687306 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.687276 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5889f46b-2aa6-4eab-a3e4-171b81b804de-istio-envoy\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:33:20.687306 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.687285 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82d84\" (UniqueName: \"kubernetes.io/projected/5889f46b-2aa6-4eab-a3e4-171b81b804de-kube-api-access-82d84\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:33:20.687306 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.687295 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5889f46b-2aa6-4eab-a3e4-171b81b804de-istio-token\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:33:20.742784 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.742754 4952 generic.go:356] "Generic (PLEG): container finished" podID="5889f46b-2aa6-4eab-a3e4-171b81b804de" containerID="a96c29f9b629337402ea621829c72e5a31b1e7ab2eb79ab1f496871cb288fc85" exitCode=0 Oct 30 23:33:20.743186 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.742796 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" Oct 30 23:33:20.743186 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.742822 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" event={"ID":"5889f46b-2aa6-4eab-a3e4-171b81b804de","Type":"ContainerDied","Data":"a96c29f9b629337402ea621829c72e5a31b1e7ab2eb79ab1f496871cb288fc85"} Oct 30 23:33:20.743186 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.742866 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm" event={"ID":"5889f46b-2aa6-4eab-a3e4-171b81b804de","Type":"ContainerDied","Data":"768a31c7d71f112216c921554ab5e581fc863033b145689d01f31d3473bec548"} Oct 30 23:33:20.743186 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.742887 4952 scope.go:122] "RemoveContainer" containerID="a96c29f9b629337402ea621829c72e5a31b1e7ab2eb79ab1f496871cb288fc85" Oct 30 23:33:20.754820 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.754803 4952 scope.go:122] "RemoveContainer" containerID="a96c29f9b629337402ea621829c72e5a31b1e7ab2eb79ab1f496871cb288fc85" Oct 30 23:33:20.755143 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:33:20.755119 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a96c29f9b629337402ea621829c72e5a31b1e7ab2eb79ab1f496871cb288fc85\": container with ID starting with a96c29f9b629337402ea621829c72e5a31b1e7ab2eb79ab1f496871cb288fc85 not found: ID does not exist" containerID="a96c29f9b629337402ea621829c72e5a31b1e7ab2eb79ab1f496871cb288fc85" Oct 30 23:33:20.755244 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.755147 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a96c29f9b629337402ea621829c72e5a31b1e7ab2eb79ab1f496871cb288fc85"} err="failed to get container status \"a96c29f9b629337402ea621829c72e5a31b1e7ab2eb79ab1f496871cb288fc85\": rpc error: code = NotFound desc = could not find container \"a96c29f9b629337402ea621829c72e5a31b1e7ab2eb79ab1f496871cb288fc85\": container with ID starting with a96c29f9b629337402ea621829c72e5a31b1e7ab2eb79ab1f496871cb288fc85 not found: ID does not exist" Oct 30 23:33:20.908956 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.908928 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm"] Oct 30 23:33:20.912948 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.912926 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-mf2zm"] Oct 30 23:33:20.972713 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:20.972652 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5889f46b-2aa6-4eab-a3e4-171b81b804de" path="/var/lib/kubelet/pods/5889f46b-2aa6-4eab-a3e4-171b81b804de/volumes" Oct 30 23:33:21.142155 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:33:21.142119 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:33:22.337769 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:22.337726 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:22.989553 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:22.989517 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:23.229243 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:23.229209 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:23.459218 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:23.459182 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:25.465137 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:25.465099 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:33:27.299981 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:27.299952 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:27.557388 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:27.557306 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:28.329223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:28.329186 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:28.556578 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:28.556532 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:31.482932 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:31.482896 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:31.707525 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:31.707493 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:33.527417 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:33.526465 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:38.518214 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:38.518168 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:38.749890 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:38.749859 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:38.974659 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:38.974628 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:42.666980 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:42.666918 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:42.686705 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:42.686680 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9"] Oct 30 23:33:42.686930 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:42.686905 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9" podUID="4212bd64-9888-4ed7-815a-0a502bfdfb6c" containerName="maas-api" containerID="cri-o://7a1235260751dd38f5124172dffc890c48058b7298153acf17638baaa8918f90" gracePeriod=30 Oct 30 23:33:42.924051 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:42.923835 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:42.984954 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:42.984835 4952 generic.go:356] "Generic (PLEG): container finished" podID="4212bd64-9888-4ed7-815a-0a502bfdfb6c" containerID="7a1235260751dd38f5124172dffc890c48058b7298153acf17638baaa8918f90" exitCode=0 Oct 30 23:33:42.991172 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:42.990597 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9" event={"ID":"4212bd64-9888-4ed7-815a-0a502bfdfb6c","Type":"ContainerDied","Data":"7a1235260751dd38f5124172dffc890c48058b7298153acf17638baaa8918f90"} Oct 30 23:33:43.141401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:43.141370 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9" Oct 30 23:33:43.252403 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:43.252317 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4212bd64-9888-4ed7-815a-0a502bfdfb6c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4212bd64-9888-4ed7-815a-0a502bfdfb6c-openshift-service-ca\") pod \"4212bd64-9888-4ed7-815a-0a502bfdfb6c\" (UID: \"4212bd64-9888-4ed7-815a-0a502bfdfb6c\") " Oct 30 23:33:43.252583 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:43.252439 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4212bd64-9888-4ed7-815a-0a502bfdfb6c-kube-api-access-vx85p\" (UniqueName: \"kubernetes.io/projected/4212bd64-9888-4ed7-815a-0a502bfdfb6c-kube-api-access-vx85p\") pod \"4212bd64-9888-4ed7-815a-0a502bfdfb6c\" (UID: \"4212bd64-9888-4ed7-815a-0a502bfdfb6c\") " Oct 30 23:33:43.252583 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:43.252472 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4212bd64-9888-4ed7-815a-0a502bfdfb6c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4212bd64-9888-4ed7-815a-0a502bfdfb6c-maas-api-tls\") pod \"4212bd64-9888-4ed7-815a-0a502bfdfb6c\" (UID: \"4212bd64-9888-4ed7-815a-0a502bfdfb6c\") " Oct 30 23:33:43.252583 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:43.252555 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4212bd64-9888-4ed7-815a-0a502bfdfb6c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4212bd64-9888-4ed7-815a-0a502bfdfb6c-metrics-certs\") pod \"4212bd64-9888-4ed7-815a-0a502bfdfb6c\" (UID: \"4212bd64-9888-4ed7-815a-0a502bfdfb6c\") " Oct 30 23:33:43.252760 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:43.252706 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4212bd64-9888-4ed7-815a-0a502bfdfb6c-openshift-service-ca" pod "4212bd64-9888-4ed7-815a-0a502bfdfb6c" (UID: "4212bd64-9888-4ed7-815a-0a502bfdfb6c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:33:43.253281 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:43.253258 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4212bd64-9888-4ed7-815a-0a502bfdfb6c-openshift-service-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:33:43.255288 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:43.255254 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4212bd64-9888-4ed7-815a-0a502bfdfb6c-metrics-certs" pod "4212bd64-9888-4ed7-815a-0a502bfdfb6c" (UID: "4212bd64-9888-4ed7-815a-0a502bfdfb6c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:33:43.257194 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:43.257144 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4212bd64-9888-4ed7-815a-0a502bfdfb6c-maas-api-tls" pod "4212bd64-9888-4ed7-815a-0a502bfdfb6c" (UID: "4212bd64-9888-4ed7-815a-0a502bfdfb6c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:33:43.259555 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:43.259468 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4212bd64-9888-4ed7-815a-0a502bfdfb6c-kube-api-access-vx85p" pod "4212bd64-9888-4ed7-815a-0a502bfdfb6c" (UID: "4212bd64-9888-4ed7-815a-0a502bfdfb6c"). InnerVolumeSpecName "kube-api-access-vx85p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:33:43.354773 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:43.354735 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4212bd64-9888-4ed7-815a-0a502bfdfb6c-metrics-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:33:43.354773 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:43.354768 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx85p\" (UniqueName: \"kubernetes.io/projected/4212bd64-9888-4ed7-815a-0a502bfdfb6c-kube-api-access-vx85p\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:33:43.354773 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:43.354782 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4212bd64-9888-4ed7-815a-0a502bfdfb6c-maas-api-tls\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:33:44.003063 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:43.999562 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9" Oct 30 23:33:44.003063 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:43.999683 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9" event={"ID":"4212bd64-9888-4ed7-815a-0a502bfdfb6c","Type":"ContainerDied","Data":"b4707d8206f8717c2386a28e937c8979a1a9935ae3914c60c5a69378336aea33"} Oct 30 23:33:44.003063 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:43.999747 4952 scope.go:122] "RemoveContainer" containerID="7a1235260751dd38f5124172dffc890c48058b7298153acf17638baaa8918f90" Oct 30 23:33:44.167329 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:44.167298 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9"] Oct 30 23:33:44.171264 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:44.171240 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5f9ec6be-5dfbc584c9-8g4v9"] Oct 30 23:33:44.800085 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:44.797710 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:44.972776 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:44.972740 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4212bd64-9888-4ed7-815a-0a502bfdfb6c" path="/var/lib/kubelet/pods/4212bd64-9888-4ed7-815a-0a502bfdfb6c/volumes" Oct 30 23:33:49.051190 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:49.051160 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15b85996-b299h"] Oct 30 23:33:49.101008 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:49.100981 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15b85996-b299h"] Oct 30 23:33:49.186206 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:49.186174 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:49.474443 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:49.474408 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:49.701313 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:49.701274 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:50.105573 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:50.105544 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:50.973461 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:50.973428 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35a5425f-7388-4b75-a39e-827ae73f8be6" path="/var/lib/kubelet/pods/35a5425f-7388-4b75-a39e-827ae73f8be6/volumes" Oct 30 23:33:51.142716 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:33:51.142680 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:33:54.708813 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:54.708768 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:56.026293 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:56.026249 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:56.263739 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:56.263708 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:56.492579 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:56.492543 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:57.847108 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:57.847071 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:58.074257 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:58.074225 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:33:59.635930 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:33:59.635891 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:03.493354 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:03.493301 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:03.730814 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:03.730784 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:04.089382 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:04.089354 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:04.425743 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:04.425664 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:04.665672 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:04.665636 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:04.980677 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:04.980645 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:34:05.365597 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:05.365552 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:05.599526 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:05.599494 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:09.500236 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:09.500193 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:12.400270 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:12.400237 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:14.282616 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:14.282582 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:18.863442 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:18.863374 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:19.129350 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:19.129267 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:19.365716 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:19.365685 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:20.072936 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:20.072901 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:21.143572 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:34:21.143535 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:34:22.050565 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:22.050533 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf56d53c-wphkc"] Oct 30 23:34:22.099871 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:22.099841 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf56d53c-wphkc"] Oct 30 23:34:22.973650 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:22.973612 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e89ad741-5b63-4713-8c79-698219b308e0" path="/var/lib/kubelet/pods/e89ad741-5b63-4713-8c79-698219b308e0/volumes" Oct 30 23:34:23.266946 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:23.266828 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:24.026188 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:24.026158 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:25.393686 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:25.393650 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:26.012879 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:26.012008 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:26.255222 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:26.255148 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:28.650701 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:28.650662 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:29.856975 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:29.856944 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:30.479291 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:30.479257 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:30.715462 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:30.715425 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:31.367255 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:31.367222 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:32.047978 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:32.047942 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:32.899029 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:32.898997 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:34.280545 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:34.280510 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:34.550301 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:34.550196 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:34.820677 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:34.820560 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:36.539345 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:36.539304 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:36.779455 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:36.779423 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:39.917291 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:39.917258 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:43.297360 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:43.297284 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:43.583456 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:43.583320 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:44.939878 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:44.939796 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:45.179536 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:45.179480 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:45.439334 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:45.439298 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:46.536066 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:46.535987 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:46.854282 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:46.854228 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:47.162176 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:47.162086 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:47.541558 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:47.541460 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:50.385541 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:50.385498 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:50.947647 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:50.947608 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:51.150616 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:34:51.150578 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:34:51.243612 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:51.243530 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:53.616558 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.616527 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2"] Oct 30 23:34:53.617209 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.617193 4952 cpu_manager.go:414] "containerMap: removing container" podUID="43ea043e-1ca0-497f-b8be-30e75effa2b0" containerName="maas-api" Oct 30 23:34:53.617273 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.617210 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43ea043e-1ca0-497f-b8be-30e75effa2b0" containerName="maas-api" Oct 30 23:34:53.617273 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.617227 4952 cpu_manager.go:414] "containerMap: removing container" podUID="5889f46b-2aa6-4eab-a3e4-171b81b804de" containerName="istio-proxy" Oct 30 23:34:53.617273 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.617232 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5889f46b-2aa6-4eab-a3e4-171b81b804de" containerName="istio-proxy" Oct 30 23:34:53.617273 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.617262 4952 cpu_manager.go:414] "containerMap: removing container" podUID="4212bd64-9888-4ed7-815a-0a502bfdfb6c" containerName="maas-api" Oct 30 23:34:53.617273 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.617268 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4212bd64-9888-4ed7-815a-0a502bfdfb6c" containerName="maas-api" Oct 30 23:34:53.617433 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.617282 4952 cpu_manager.go:414] "containerMap: removing container" podUID="4863fa66-8c17-49c8-be19-01693e905cd3" containerName="maas-api" Oct 30 23:34:53.617433 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.617287 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4863fa66-8c17-49c8-be19-01693e905cd3" containerName="maas-api" Oct 30 23:34:53.617433 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.617378 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="4863fa66-8c17-49c8-be19-01693e905cd3" containerName="maas-api" Oct 30 23:34:53.617433 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.617387 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="43ea043e-1ca0-497f-b8be-30e75effa2b0" containerName="maas-api" Oct 30 23:34:53.617433 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.617395 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="5889f46b-2aa6-4eab-a3e4-171b81b804de" containerName="istio-proxy" Oct 30 23:34:53.617433 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.617407 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="4212bd64-9888-4ed7-815a-0a502bfdfb6c" containerName="maas-api" Oct 30 23:34:53.621215 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.621195 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" Oct 30 23:34:53.673517 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.673494 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-55666687-openshift-default-dockercfg-8kmlb\"" Oct 30 23:34:53.728184 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.728155 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2"] Oct 30 23:34:53.763760 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.763733 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a365be8c-838f-46e0-a439-00700619a558-istio-token\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-785d2\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" Oct 30 23:34:53.763919 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.763780 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9f55\" (UniqueName: \"kubernetes.io/projected/a365be8c-838f-46e0-a439-00700619a558-kube-api-access-w9f55\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-785d2\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" Oct 30 23:34:53.763919 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.763849 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a365be8c-838f-46e0-a439-00700619a558-istio-data\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-785d2\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" Oct 30 23:34:53.763919 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.763866 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a365be8c-838f-46e0-a439-00700619a558-istio-podinfo\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-785d2\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" Oct 30 23:34:53.764409 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.763926 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a365be8c-838f-46e0-a439-00700619a558-workload-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-785d2\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" Oct 30 23:34:53.764409 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.763965 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a365be8c-838f-46e0-a439-00700619a558-workload-certs\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-785d2\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" Oct 30 23:34:53.764409 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.763998 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a365be8c-838f-46e0-a439-00700619a558-istiod-ca-cert\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-785d2\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" Oct 30 23:34:53.764409 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.764029 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a365be8c-838f-46e0-a439-00700619a558-credential-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-785d2\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" Oct 30 23:34:53.764409 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.764084 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a365be8c-838f-46e0-a439-00700619a558-istio-envoy\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-785d2\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" Oct 30 23:34:53.865541 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.865506 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a365be8c-838f-46e0-a439-00700619a558-workload-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-785d2\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" Oct 30 23:34:53.865729 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.865568 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a365be8c-838f-46e0-a439-00700619a558-workload-certs\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-785d2\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" Oct 30 23:34:53.865729 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.865602 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a365be8c-838f-46e0-a439-00700619a558-istiod-ca-cert\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-785d2\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" Oct 30 23:34:53.865729 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.865644 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a365be8c-838f-46e0-a439-00700619a558-credential-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-785d2\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" Oct 30 23:34:53.865729 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.865690 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a365be8c-838f-46e0-a439-00700619a558-istio-envoy\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-785d2\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" Oct 30 23:34:53.865962 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.865785 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a365be8c-838f-46e0-a439-00700619a558-istio-token\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-785d2\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" Oct 30 23:34:53.865962 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.865825 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9f55\" (UniqueName: \"kubernetes.io/projected/a365be8c-838f-46e0-a439-00700619a558-kube-api-access-w9f55\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-785d2\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" Oct 30 23:34:53.865962 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.865915 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a365be8c-838f-46e0-a439-00700619a558-istio-data\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-785d2\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" Oct 30 23:34:53.865962 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.865931 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a365be8c-838f-46e0-a439-00700619a558-workload-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-785d2\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" Oct 30 23:34:53.865962 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.865941 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a365be8c-838f-46e0-a439-00700619a558-istio-podinfo\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-785d2\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" Oct 30 23:34:53.866747 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.866396 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a365be8c-838f-46e0-a439-00700619a558-istiod-ca-cert\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-785d2\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" Oct 30 23:34:53.866747 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.866648 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a365be8c-838f-46e0-a439-00700619a558-credential-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-785d2\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" Oct 30 23:34:53.866747 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.866674 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a365be8c-838f-46e0-a439-00700619a558-istio-data\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-785d2\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" Oct 30 23:34:53.867304 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.867279 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a365be8c-838f-46e0-a439-00700619a558-workload-certs\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-785d2\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" Oct 30 23:34:53.868650 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.868625 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a365be8c-838f-46e0-a439-00700619a558-istio-envoy\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-785d2\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" Oct 30 23:34:53.869264 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.869243 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a365be8c-838f-46e0-a439-00700619a558-istio-podinfo\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-785d2\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" Oct 30 23:34:53.924399 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.924361 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a365be8c-838f-46e0-a439-00700619a558-istio-token\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-785d2\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" Oct 30 23:34:53.925395 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.925369 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9f55\" (UniqueName: \"kubernetes.io/projected/a365be8c-838f-46e0-a439-00700619a558-kube-api-access-w9f55\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-785d2\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" Oct 30 23:34:53.935929 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:53.935899 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" Oct 30 23:34:54.366899 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:54.366876 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2"] Oct 30 23:34:54.367446 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:34:54.367420 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda365be8c_838f_46e0_a439_00700619a558.slice/crio-edbe0239c8311685d1c540e4013a54517b6be42424c28df617b67da1d90a1582 WatchSource:0}: Error finding container edbe0239c8311685d1c540e4013a54517b6be42424c28df617b67da1d90a1582: Status 404 returned error can't find the container with id edbe0239c8311685d1c540e4013a54517b6be42424c28df617b67da1d90a1582 Oct 30 23:34:54.370106 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:54.370068 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:34:54.370223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:54.370149 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:34:54.370223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:54.370200 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:34:54.712424 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:54.712347 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" event={"ID":"a365be8c-838f-46e0-a439-00700619a558","Type":"ContainerStarted","Data":"7f377bc9dec227a8d54c09e6ab0cc1bd3b94ffd7db12f3626d7ddf130245db28"} Oct 30 23:34:54.712424 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:54.712410 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" event={"ID":"a365be8c-838f-46e0-a439-00700619a558","Type":"ContainerStarted","Data":"edbe0239c8311685d1c540e4013a54517b6be42424c28df617b67da1d90a1582"} Oct 30 23:34:54.833557 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:54.833504 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" podStartSLOduration=1.833489208 podStartE2EDuration="1.833489208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:34:54.829605746 +0000 UTC m=+1713.973435979" watchObservedRunningTime="2026-10-30 23:34:54.833489208 +0000 UTC m=+1713.977319412" Oct 30 23:34:54.936239 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:54.936199 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" Oct 30 23:34:55.687837 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:55.687760 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:34:55.938912 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:55.938835 4952 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-785d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 23:34:55.938912 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:55.938885 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" podUID="a365be8c-838f-46e0-a439-00700619a558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 23:34:56.937073 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:56.937006 4952 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-785d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 23:34:56.937296 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:56.937096 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" podUID="a365be8c-838f-46e0-a439-00700619a558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 23:34:57.936472 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:57.936442 4952 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-785d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 23:34:57.936931 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:57.936495 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" podUID="a365be8c-838f-46e0-a439-00700619a558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 23:34:58.937137 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:58.937097 4952 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-785d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 23:34:58.937643 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:58.937183 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" podUID="a365be8c-838f-46e0-a439-00700619a558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 23:34:59.937127 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:59.937085 4952 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-785d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 23:34:59.937343 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:34:59.937174 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" podUID="a365be8c-838f-46e0-a439-00700619a558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 23:35:00.941057 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:00.941014 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" Oct 30 23:35:00.941473 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:00.941360 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" Oct 30 23:35:00.942000 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:00.941983 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" Oct 30 23:35:00.989437 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:00.989405 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:35:03.879266 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:03.879232 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:35:04.123411 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:04.123376 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:35:06.371706 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:06.371651 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:35:11.375180 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:11.375147 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:35:15.504852 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:15.504813 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:35:16.181729 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:16.181698 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:35:20.393749 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:20.393699 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:35:20.644232 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:20.644133 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:35:21.144446 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:35:21.144409 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:35:21.364122 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:21.364092 4952 scope.go:122] "RemoveContainer" containerID="f51bd76e1e584bfd6e696cb683cbc83b6c9afe982a6d0702b83d212e364fcc6f" Oct 30 23:35:21.384823 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:21.384783 4952 scope.go:122] "RemoveContainer" containerID="3fc139d4ffeea9682f0023467f174bedbaca0a60142b2be651a6e23e720dc534" Oct 30 23:35:22.009359 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:22.009328 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:35:22.327306 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:22.327169 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:35:22.584964 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:22.584873 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:35:22.828379 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:22.828348 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:35:23.469195 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:23.469166 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:35:24.269421 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:24.269386 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:35:24.742357 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:24.742321 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:35:24.987542 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:24.987509 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:35:27.517579 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:27.517543 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:35:27.755453 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:27.755421 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:35:27.992803 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:27.992745 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:35:28.255149 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:28.255016 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:35:28.492682 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:28.492646 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:35:32.281367 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:32.281291 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc"] Oct 30 23:35:32.281783 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:32.281515 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" podUID="c0e935ae-8758-4f00-beca-d6b4c7c38dd6" containerName="istio-proxy" containerID="cri-o://0bd97141d47ac3b35cd21541f673920f3ac09d5c0b7960dbc078ecff772775cc" gracePeriod=30 Oct 30 23:35:32.997000 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:32.996952 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:35:33.254821 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:33.254717 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:35:33.488803 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:33.488769 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:35:37.052672 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:37.052641 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 23:35:37.053093 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:37.052689 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" podUID="c0e935ae-8758-4f00-beca-d6b4c7c38dd6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:35:37.713578 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:37.713538 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:35:37.942374 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:37.942338 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:35:38.047532 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.047500 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" Oct 30 23:35:38.067310 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.067280 4952 generic.go:356] "Generic (PLEG): container finished" podID="c0e935ae-8758-4f00-beca-d6b4c7c38dd6" containerID="0bd97141d47ac3b35cd21541f673920f3ac09d5c0b7960dbc078ecff772775cc" exitCode=0 Oct 30 23:35:38.067722 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.067324 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" event={"ID":"c0e935ae-8758-4f00-beca-d6b4c7c38dd6","Type":"ContainerDied","Data":"0bd97141d47ac3b35cd21541f673920f3ac09d5c0b7960dbc078ecff772775cc"} Oct 30 23:35:38.067722 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.067340 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" Oct 30 23:35:38.067722 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.067365 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc" event={"ID":"c0e935ae-8758-4f00-beca-d6b4c7c38dd6","Type":"ContainerDied","Data":"168d2f53c0591b6ae572b9913fe2da9723da78dc470fd3951c91f299376904b7"} Oct 30 23:35:38.067722 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.067388 4952 scope.go:122] "RemoveContainer" containerID="0bd97141d47ac3b35cd21541f673920f3ac09d5c0b7960dbc078ecff772775cc" Oct 30 23:35:38.078057 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.078022 4952 scope.go:122] "RemoveContainer" containerID="0bd97141d47ac3b35cd21541f673920f3ac09d5c0b7960dbc078ecff772775cc" Oct 30 23:35:38.078337 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:35:38.078316 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bd97141d47ac3b35cd21541f673920f3ac09d5c0b7960dbc078ecff772775cc\": container with ID starting with 0bd97141d47ac3b35cd21541f673920f3ac09d5c0b7960dbc078ecff772775cc not found: ID does not exist" containerID="0bd97141d47ac3b35cd21541f673920f3ac09d5c0b7960dbc078ecff772775cc" Oct 30 23:35:38.078386 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.078347 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bd97141d47ac3b35cd21541f673920f3ac09d5c0b7960dbc078ecff772775cc"} err="failed to get container status \"0bd97141d47ac3b35cd21541f673920f3ac09d5c0b7960dbc078ecff772775cc\": rpc error: code = NotFound desc = could not find container \"0bd97141d47ac3b35cd21541f673920f3ac09d5c0b7960dbc078ecff772775cc\": container with ID starting with 0bd97141d47ac3b35cd21541f673920f3ac09d5c0b7960dbc078ecff772775cc not found: ID does not exist" Oct 30 23:35:38.102892 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.102867 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-istio-podinfo\") pod \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " Oct 30 23:35:38.102996 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.102898 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-istiod-ca-cert\") pod \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " Oct 30 23:35:38.102996 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.102958 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-istio-data\") pod \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " Oct 30 23:35:38.102996 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.102977 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-istio-token\" (UniqueName: \"kubernetes.io/projected/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-istio-token\") pod \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " Oct 30 23:35:38.103171 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.103012 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-workload-certs\") pod \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " Oct 30 23:35:38.103228 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.103200 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-credential-socket\") pod \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " Oct 30 23:35:38.103279 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.103262 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-kube-api-access-r5xq8\" (UniqueName: \"kubernetes.io/projected/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-kube-api-access-r5xq8\") pod \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " Oct 30 23:35:38.103331 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.103295 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-istio-envoy\") pod \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " Oct 30 23:35:38.103385 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.103331 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-workload-socket\") pod \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\" (UID: \"c0e935ae-8758-4f00-beca-d6b4c7c38dd6\") " Oct 30 23:35:38.103385 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.103330 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-istio-data" pod "c0e935ae-8758-4f00-beca-d6b4c7c38dd6" (UID: "c0e935ae-8758-4f00-beca-d6b4c7c38dd6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:35:38.103492 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.103388 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-workload-certs" pod "c0e935ae-8758-4f00-beca-d6b4c7c38dd6" (UID: "c0e935ae-8758-4f00-beca-d6b4c7c38dd6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:35:38.103550 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.103477 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-credential-socket" pod "c0e935ae-8758-4f00-beca-d6b4c7c38dd6" (UID: "c0e935ae-8758-4f00-beca-d6b4c7c38dd6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:35:38.103550 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.103527 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-istiod-ca-cert" pod "c0e935ae-8758-4f00-beca-d6b4c7c38dd6" (UID: "c0e935ae-8758-4f00-beca-d6b4c7c38dd6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:35:38.103650 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.103635 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-workload-socket" pod "c0e935ae-8758-4f00-beca-d6b4c7c38dd6" (UID: "c0e935ae-8758-4f00-beca-d6b4c7c38dd6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:35:38.104375 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.104352 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-credential-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:35:38.104480 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.104382 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-workload-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:35:38.104480 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.104398 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-istiod-ca-cert\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:35:38.104480 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.104415 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-istio-data\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:35:38.104480 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.104430 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-workload-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:35:38.105396 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.105373 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-istio-token" pod "c0e935ae-8758-4f00-beca-d6b4c7c38dd6" (UID: "c0e935ae-8758-4f00-beca-d6b4c7c38dd6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:35:38.105619 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.105589 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-kube-api-access-r5xq8" pod "c0e935ae-8758-4f00-beca-d6b4c7c38dd6" (UID: "c0e935ae-8758-4f00-beca-d6b4c7c38dd6"). InnerVolumeSpecName "kube-api-access-r5xq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:35:38.105879 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.105857 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-istio-envoy" pod "c0e935ae-8758-4f00-beca-d6b4c7c38dd6" (UID: "c0e935ae-8758-4f00-beca-d6b4c7c38dd6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:35:38.106004 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.105980 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-istio-podinfo" pod "c0e935ae-8758-4f00-beca-d6b4c7c38dd6" (UID: "c0e935ae-8758-4f00-beca-d6b4c7c38dd6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:35:38.173538 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.173505 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:35:38.206007 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.205975 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5xq8\" (UniqueName: \"kubernetes.io/projected/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-kube-api-access-r5xq8\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:35:38.206007 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.206000 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-istio-envoy\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:35:38.206007 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.206010 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-istio-podinfo\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:35:38.206196 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.206019 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e935ae-8758-4f00-beca-d6b4c7c38dd6-istio-token\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:35:38.492967 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.492935 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc"] Oct 30 23:35:38.499703 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.499673 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-hbwzc"] Oct 30 23:35:38.973407 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:38.973375 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0e935ae-8758-4f00-beca-d6b4c7c38dd6" path="/var/lib/kubelet/pods/c0e935ae-8758-4f00-beca-d6b4c7c38dd6/volumes" Oct 30 23:35:42.761544 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:42.761514 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:35:47.680317 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:47.680277 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:35:51.145182 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:35:51.145146 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:35:52.516513 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:52.516481 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:35:52.777592 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:52.777498 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:35:57.682910 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:35:57.682857 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:36:02.515767 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:02.515722 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:36:04.861708 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:04.861677 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:36:07.344185 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:07.344147 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:36:11.099743 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:11.099708 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:36:12.200757 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:12.200662 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:36:14.972832 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:14.972803 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:36:17.265544 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:17.265504 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:36:20.741398 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:20.741353 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:36:21.148029 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:36:21.147985 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:36:21.241191 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:21.241159 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft_91651ce2-4673-4ced-b26d-e94ff8829d11/extract/0.log" Oct 30 23:36:21.251132 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:21.251108 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft_91651ce2-4673-4ced-b26d-e94ff8829d11/extract/0.log" Oct 30 23:36:21.253336 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:21.253315 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/2.log" Oct 30 23:36:21.260874 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:21.260857 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 30 23:36:21.264160 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:21.264133 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/2.log" Oct 30 23:36:21.270159 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:21.270143 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 30 23:36:22.211350 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:22.211299 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:36:27.063509 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:27.063473 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:36:31.810351 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:31.810313 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:36:33.602028 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:33.601979 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:36:36.769173 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:36.768979 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:36:41.798878 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:41.798839 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:36:43.302346 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:43.302314 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:36:43.709404 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:43.709364 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:36:44.783366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:44.783264 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:36:45.012223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:45.012175 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:36:46.992406 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:46.992371 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:36:49.157553 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:49.157509 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:36:49.427783 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:49.427674 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:36:49.683447 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:49.683351 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:36:51.163170 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:36:51.163133 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:36:52.613133 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:52.613076 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:36:54.682441 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:54.682403 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:36:55.381340 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:55.381297 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:36:56.273064 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:56.270934 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:36:56.723701 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:56.723665 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:36:58.169171 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:58.169135 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:36:58.847061 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:58.846063 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:36:59.103857 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:36:59.103771 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:01.131535 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:01.131446 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:01.495661 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:01.495585 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:01.816107 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:01.813493 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:02.056622 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:02.054671 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:02.295393 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:02.295356 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:02.710365 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:02.710323 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:03.692429 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:03.692397 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:07.039935 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:07.039903 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:07.283157 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:07.283118 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:08.698848 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:08.698815 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:08.956223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:08.956124 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:09.194306 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:09.194273 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:13.559601 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:13.559569 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:13.787322 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:13.787283 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:14.085104 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:14.085030 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:14.319339 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:14.319310 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:19.070228 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:19.070188 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:21.068137 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:21.068105 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:21.143099 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:37:21.143059 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:37:21.316778 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:21.316742 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:23.824059 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:23.823992 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:28.596630 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:28.596578 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:29.811070 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:29.811009 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:30.471158 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:30.471121 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:31.079753 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:31.079713 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:37:31.230150 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:31.230107 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:33.508670 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:33.508636 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:35.005946 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:35.005880 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:35.257481 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:35.257405 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:35.500319 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:35.500271 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:38.401903 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:38.401866 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:38.634001 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:38.633963 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:38.866231 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:38.866186 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:43.692602 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:43.692554 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:45.794385 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:45.794339 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:46.087765 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:46.087726 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:48.578498 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:48.578462 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:51.143599 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:37:51.143562 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:37:51.403362 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:51.403281 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:51.797764 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:51.797646 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:52.496084 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:52.496002 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:53.177302 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:53.177260 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:53.836400 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:53.836369 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:57.287344 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:57.287308 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:57.522399 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:57.522367 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:37:59.059452 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:37:59.059413 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:38:00.544741 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:00.544668 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:38:00.773307 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:00.773273 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:38:02.883419 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:02.883383 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz"] Oct 30 23:38:02.883814 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:02.883635 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" podUID="0646012c-55f4-4e2b-8dc8-dcd9603504a3" containerName="istio-proxy" containerID="cri-o://d0c3fb9c24656cb620d1a370bb4b5a63286fcd56136944b52989b158225c6d40" gracePeriod=30 Oct 30 23:38:02.933292 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:02.933263 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq"] Oct 30 23:38:02.933529 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:02.933490 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" podUID="293e802c-8e13-4e98-9026-8f13ee239f58" containerName="istio-proxy" containerID="cri-o://c0d6e0408408a3d44f7780b7f4ed0083b45b700ed3b191b16ce20395de32ca13" gracePeriod=30 Oct 30 23:38:03.084928 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:03.084895 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg"] Oct 30 23:38:03.085189 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:03.085153 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" podUID="824cbd14-4a34-4df1-864e-63bb34900aab" containerName="istio-proxy" containerID="cri-o://18ad8f0ed81d0e16352354e96f03c6c0defa7d402c7c11102f4f0aaa88ee4b5d" gracePeriod=30 Oct 30 23:38:04.159116 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:04.159080 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj"] Oct 30 23:38:04.159977 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:04.159939 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" podUID="482b6308-b55b-445f-b1a4-f95bab24f844" containerName="istio-proxy" containerID="cri-o://f09fd34c21cdd17cb1aa3ad23dc79b4840838bc9b358c9566c7fd598ffdabeee" gracePeriod=30 Oct 30 23:38:04.189837 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:04.189745 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:38:04.413164 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:04.413081 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:38:04.633072 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:04.632988 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:38:04.870103 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:04.870069 4952 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 23:38:04.870299 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:04.870142 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" podUID="293e802c-8e13-4e98-9026-8f13ee239f58" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:38:05.937341 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:05.937308 4952 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 23:38:05.937741 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:05.937369 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" podUID="824cbd14-4a34-4df1-864e-63bb34900aab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:38:07.450276 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:07.450241 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:38:08.215956 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.215931 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" Oct 30 23:38:08.221580 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.221549 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" Oct 30 23:38:08.302682 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.302646 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0646012c-55f4-4e2b-8dc8-dcd9603504a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0646012c-55f4-4e2b-8dc8-dcd9603504a3-workload-socket\") pod \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " Oct 30 23:38:08.302844 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.302729 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0646012c-55f4-4e2b-8dc8-dcd9603504a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0646012c-55f4-4e2b-8dc8-dcd9603504a3-credential-socket\") pod \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " Oct 30 23:38:08.302844 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.302791 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/293e802c-8e13-4e98-9026-8f13ee239f58-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/293e802c-8e13-4e98-9026-8f13ee239f58-istiod-ca-cert\") pod \"293e802c-8e13-4e98-9026-8f13ee239f58\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " Oct 30 23:38:08.302966 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.302877 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0646012c-55f4-4e2b-8dc8-dcd9603504a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0646012c-55f4-4e2b-8dc8-dcd9603504a3-workload-certs\") pod \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " Oct 30 23:38:08.302966 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.302906 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/293e802c-8e13-4e98-9026-8f13ee239f58-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/293e802c-8e13-4e98-9026-8f13ee239f58-istio-envoy\") pod \"293e802c-8e13-4e98-9026-8f13ee239f58\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " Oct 30 23:38:08.302966 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.302936 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0646012c-55f4-4e2b-8dc8-dcd9603504a3-kube-api-access-jd75n\" (UniqueName: \"kubernetes.io/projected/0646012c-55f4-4e2b-8dc8-dcd9603504a3-kube-api-access-jd75n\") pod \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " Oct 30 23:38:08.303256 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.302994 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0646012c-55f4-4e2b-8dc8-dcd9603504a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0646012c-55f4-4e2b-8dc8-dcd9603504a3-istio-podinfo\") pod \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " Oct 30 23:38:08.303256 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.303025 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/293e802c-8e13-4e98-9026-8f13ee239f58-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/293e802c-8e13-4e98-9026-8f13ee239f58-workload-socket\") pod \"293e802c-8e13-4e98-9026-8f13ee239f58\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " Oct 30 23:38:08.303256 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.303094 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/293e802c-8e13-4e98-9026-8f13ee239f58-istio-token\" (UniqueName: \"kubernetes.io/projected/293e802c-8e13-4e98-9026-8f13ee239f58-istio-token\") pod \"293e802c-8e13-4e98-9026-8f13ee239f58\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " Oct 30 23:38:08.303256 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.303211 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0646012c-55f4-4e2b-8dc8-dcd9603504a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0646012c-55f4-4e2b-8dc8-dcd9603504a3-istio-envoy\") pod \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " Oct 30 23:38:08.303256 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.303245 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0646012c-55f4-4e2b-8dc8-dcd9603504a3-workload-certs" pod "0646012c-55f4-4e2b-8dc8-dcd9603504a3" (UID: "0646012c-55f4-4e2b-8dc8-dcd9603504a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:38:08.303256 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.303244 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/293e802c-8e13-4e98-9026-8f13ee239f58-istiod-ca-cert" pod "293e802c-8e13-4e98-9026-8f13ee239f58" (UID: "293e802c-8e13-4e98-9026-8f13ee239f58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:38:08.303567 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.303250 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0646012c-55f4-4e2b-8dc8-dcd9603504a3-istio-token\" (UniqueName: \"kubernetes.io/projected/0646012c-55f4-4e2b-8dc8-dcd9603504a3-istio-token\") pod \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " Oct 30 23:38:08.303567 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.303304 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/293e802c-8e13-4e98-9026-8f13ee239f58-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/293e802c-8e13-4e98-9026-8f13ee239f58-workload-certs\") pod \"293e802c-8e13-4e98-9026-8f13ee239f58\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " Oct 30 23:38:08.303567 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.303360 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/293e802c-8e13-4e98-9026-8f13ee239f58-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/293e802c-8e13-4e98-9026-8f13ee239f58-istio-data\") pod \"293e802c-8e13-4e98-9026-8f13ee239f58\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " Oct 30 23:38:08.303567 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.303398 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/293e802c-8e13-4e98-9026-8f13ee239f58-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/293e802c-8e13-4e98-9026-8f13ee239f58-credential-socket\") pod \"293e802c-8e13-4e98-9026-8f13ee239f58\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " Oct 30 23:38:08.303567 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.303446 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0646012c-55f4-4e2b-8dc8-dcd9603504a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0646012c-55f4-4e2b-8dc8-dcd9603504a3-istio-data\") pod \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " Oct 30 23:38:08.303567 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.303537 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0646012c-55f4-4e2b-8dc8-dcd9603504a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0646012c-55f4-4e2b-8dc8-dcd9603504a3-istiod-ca-cert\") pod \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\" (UID: \"0646012c-55f4-4e2b-8dc8-dcd9603504a3\") " Oct 30 23:38:08.303881 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.303672 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/293e802c-8e13-4e98-9026-8f13ee239f58-kube-api-access-m4rqp\" (UniqueName: \"kubernetes.io/projected/293e802c-8e13-4e98-9026-8f13ee239f58-kube-api-access-m4rqp\") pod \"293e802c-8e13-4e98-9026-8f13ee239f58\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " Oct 30 23:38:08.303881 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.303711 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/293e802c-8e13-4e98-9026-8f13ee239f58-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/293e802c-8e13-4e98-9026-8f13ee239f58-istio-podinfo\") pod \"293e802c-8e13-4e98-9026-8f13ee239f58\" (UID: \"293e802c-8e13-4e98-9026-8f13ee239f58\") " Oct 30 23:38:08.305121 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.304351 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/293e802c-8e13-4e98-9026-8f13ee239f58-credential-socket" pod "293e802c-8e13-4e98-9026-8f13ee239f58" (UID: "293e802c-8e13-4e98-9026-8f13ee239f58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:38:08.305121 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.304680 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/293e802c-8e13-4e98-9026-8f13ee239f58-workload-socket" pod "293e802c-8e13-4e98-9026-8f13ee239f58" (UID: "293e802c-8e13-4e98-9026-8f13ee239f58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:38:08.305121 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.304751 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/293e802c-8e13-4e98-9026-8f13ee239f58-istiod-ca-cert\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:08.305121 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.304776 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0646012c-55f4-4e2b-8dc8-dcd9603504a3-workload-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:08.305121 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.304793 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/293e802c-8e13-4e98-9026-8f13ee239f58-credential-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:08.305121 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.304831 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0646012c-55f4-4e2b-8dc8-dcd9603504a3-workload-socket" pod "0646012c-55f4-4e2b-8dc8-dcd9603504a3" (UID: "0646012c-55f4-4e2b-8dc8-dcd9603504a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:38:08.305121 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.304842 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0646012c-55f4-4e2b-8dc8-dcd9603504a3-istiod-ca-cert" pod "0646012c-55f4-4e2b-8dc8-dcd9603504a3" (UID: "0646012c-55f4-4e2b-8dc8-dcd9603504a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:38:08.305121 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.305069 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0646012c-55f4-4e2b-8dc8-dcd9603504a3-credential-socket" pod "0646012c-55f4-4e2b-8dc8-dcd9603504a3" (UID: "0646012c-55f4-4e2b-8dc8-dcd9603504a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:38:08.305121 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.305082 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/293e802c-8e13-4e98-9026-8f13ee239f58-workload-certs" pod "293e802c-8e13-4e98-9026-8f13ee239f58" (UID: "293e802c-8e13-4e98-9026-8f13ee239f58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:38:08.305679 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.305147 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0646012c-55f4-4e2b-8dc8-dcd9603504a3-istio-data" pod "0646012c-55f4-4e2b-8dc8-dcd9603504a3" (UID: "0646012c-55f4-4e2b-8dc8-dcd9603504a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:38:08.305679 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.305436 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/293e802c-8e13-4e98-9026-8f13ee239f58-istio-data" pod "293e802c-8e13-4e98-9026-8f13ee239f58" (UID: "293e802c-8e13-4e98-9026-8f13ee239f58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:38:08.308277 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.308254 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0646012c-55f4-4e2b-8dc8-dcd9603504a3-istio-podinfo" pod "0646012c-55f4-4e2b-8dc8-dcd9603504a3" (UID: "0646012c-55f4-4e2b-8dc8-dcd9603504a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:38:08.308950 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.308923 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/293e802c-8e13-4e98-9026-8f13ee239f58-istio-token" pod "293e802c-8e13-4e98-9026-8f13ee239f58" (UID: "293e802c-8e13-4e98-9026-8f13ee239f58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:38:08.309760 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.309736 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0646012c-55f4-4e2b-8dc8-dcd9603504a3-istio-token" pod "0646012c-55f4-4e2b-8dc8-dcd9603504a3" (UID: "0646012c-55f4-4e2b-8dc8-dcd9603504a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:38:08.310400 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.310374 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/293e802c-8e13-4e98-9026-8f13ee239f58-istio-envoy" pod "293e802c-8e13-4e98-9026-8f13ee239f58" (UID: "293e802c-8e13-4e98-9026-8f13ee239f58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:38:08.310400 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.310393 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0646012c-55f4-4e2b-8dc8-dcd9603504a3-kube-api-access-jd75n" pod "0646012c-55f4-4e2b-8dc8-dcd9603504a3" (UID: "0646012c-55f4-4e2b-8dc8-dcd9603504a3"). InnerVolumeSpecName "kube-api-access-jd75n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:38:08.310567 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.310440 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/293e802c-8e13-4e98-9026-8f13ee239f58-istio-podinfo" pod "293e802c-8e13-4e98-9026-8f13ee239f58" (UID: "293e802c-8e13-4e98-9026-8f13ee239f58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:38:08.311945 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.311921 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0646012c-55f4-4e2b-8dc8-dcd9603504a3-istio-envoy" pod "0646012c-55f4-4e2b-8dc8-dcd9603504a3" (UID: "0646012c-55f4-4e2b-8dc8-dcd9603504a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:38:08.312022 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.311996 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/293e802c-8e13-4e98-9026-8f13ee239f58-kube-api-access-m4rqp" pod "293e802c-8e13-4e98-9026-8f13ee239f58" (UID: "293e802c-8e13-4e98-9026-8f13ee239f58"). InnerVolumeSpecName "kube-api-access-m4rqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:38:08.351412 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.350829 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 23:38:08.351412 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.350902 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" podUID="482b6308-b55b-445f-b1a4-f95bab24f844" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:38:08.389979 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.389949 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" Oct 30 23:38:08.406361 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.406331 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/293e802c-8e13-4e98-9026-8f13ee239f58-istio-envoy\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:08.406361 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.406360 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd75n\" (UniqueName: \"kubernetes.io/projected/0646012c-55f4-4e2b-8dc8-dcd9603504a3-kube-api-access-jd75n\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:08.406532 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.406370 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0646012c-55f4-4e2b-8dc8-dcd9603504a3-istio-podinfo\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:08.406532 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.406379 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/293e802c-8e13-4e98-9026-8f13ee239f58-workload-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:08.406532 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.406388 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/293e802c-8e13-4e98-9026-8f13ee239f58-istio-token\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:08.406532 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.406397 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0646012c-55f4-4e2b-8dc8-dcd9603504a3-istio-envoy\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:08.406532 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.406404 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0646012c-55f4-4e2b-8dc8-dcd9603504a3-istio-token\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:08.406532 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.406412 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/293e802c-8e13-4e98-9026-8f13ee239f58-workload-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:08.406532 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.406421 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/293e802c-8e13-4e98-9026-8f13ee239f58-istio-data\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:08.406532 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.406430 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0646012c-55f4-4e2b-8dc8-dcd9603504a3-istio-data\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:08.406532 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.406438 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0646012c-55f4-4e2b-8dc8-dcd9603504a3-istiod-ca-cert\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:08.406532 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.406447 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4rqp\" (UniqueName: \"kubernetes.io/projected/293e802c-8e13-4e98-9026-8f13ee239f58-kube-api-access-m4rqp\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:08.406532 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.406455 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/293e802c-8e13-4e98-9026-8f13ee239f58-istio-podinfo\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:08.406532 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.406464 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0646012c-55f4-4e2b-8dc8-dcd9603504a3-workload-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:08.406532 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.406473 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0646012c-55f4-4e2b-8dc8-dcd9603504a3-credential-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:08.507850 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.507772 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/824cbd14-4a34-4df1-864e-63bb34900aab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/824cbd14-4a34-4df1-864e-63bb34900aab-istio-podinfo\") pod \"824cbd14-4a34-4df1-864e-63bb34900aab\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " Oct 30 23:38:08.508352 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.507847 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/824cbd14-4a34-4df1-864e-63bb34900aab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/824cbd14-4a34-4df1-864e-63bb34900aab-credential-socket\") pod \"824cbd14-4a34-4df1-864e-63bb34900aab\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " Oct 30 23:38:08.508352 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.507907 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/824cbd14-4a34-4df1-864e-63bb34900aab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/824cbd14-4a34-4df1-864e-63bb34900aab-istiod-ca-cert\") pod \"824cbd14-4a34-4df1-864e-63bb34900aab\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " Oct 30 23:38:08.508352 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.507946 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/824cbd14-4a34-4df1-864e-63bb34900aab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/824cbd14-4a34-4df1-864e-63bb34900aab-istio-data\") pod \"824cbd14-4a34-4df1-864e-63bb34900aab\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " Oct 30 23:38:08.508352 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.507989 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/824cbd14-4a34-4df1-864e-63bb34900aab-istio-token\" (UniqueName: \"kubernetes.io/projected/824cbd14-4a34-4df1-864e-63bb34900aab-istio-token\") pod \"824cbd14-4a34-4df1-864e-63bb34900aab\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " Oct 30 23:38:08.508352 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.508100 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/824cbd14-4a34-4df1-864e-63bb34900aab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/824cbd14-4a34-4df1-864e-63bb34900aab-istio-envoy\") pod \"824cbd14-4a34-4df1-864e-63bb34900aab\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " Oct 30 23:38:08.508352 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.508136 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/824cbd14-4a34-4df1-864e-63bb34900aab-kube-api-access-wfp4z\" (UniqueName: \"kubernetes.io/projected/824cbd14-4a34-4df1-864e-63bb34900aab-kube-api-access-wfp4z\") pod \"824cbd14-4a34-4df1-864e-63bb34900aab\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " Oct 30 23:38:08.508352 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.508185 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/824cbd14-4a34-4df1-864e-63bb34900aab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/824cbd14-4a34-4df1-864e-63bb34900aab-workload-socket\") pod \"824cbd14-4a34-4df1-864e-63bb34900aab\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " Oct 30 23:38:08.508352 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.508242 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824cbd14-4a34-4df1-864e-63bb34900aab-credential-socket" pod "824cbd14-4a34-4df1-864e-63bb34900aab" (UID: "824cbd14-4a34-4df1-864e-63bb34900aab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:38:08.508751 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.508427 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/824cbd14-4a34-4df1-864e-63bb34900aab-istiod-ca-cert" pod "824cbd14-4a34-4df1-864e-63bb34900aab" (UID: "824cbd14-4a34-4df1-864e-63bb34900aab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:38:08.508751 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.508464 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/824cbd14-4a34-4df1-864e-63bb34900aab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/824cbd14-4a34-4df1-864e-63bb34900aab-workload-certs\") pod \"824cbd14-4a34-4df1-864e-63bb34900aab\" (UID: \"824cbd14-4a34-4df1-864e-63bb34900aab\") " Oct 30 23:38:08.508751 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.508641 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824cbd14-4a34-4df1-864e-63bb34900aab-workload-socket" pod "824cbd14-4a34-4df1-864e-63bb34900aab" (UID: "824cbd14-4a34-4df1-864e-63bb34900aab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:38:08.508925 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.508832 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824cbd14-4a34-4df1-864e-63bb34900aab-workload-certs" pod "824cbd14-4a34-4df1-864e-63bb34900aab" (UID: "824cbd14-4a34-4df1-864e-63bb34900aab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:38:08.508925 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.508891 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824cbd14-4a34-4df1-864e-63bb34900aab-istio-data" pod "824cbd14-4a34-4df1-864e-63bb34900aab" (UID: "824cbd14-4a34-4df1-864e-63bb34900aab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:38:08.509456 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.509425 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/824cbd14-4a34-4df1-864e-63bb34900aab-workload-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:08.509579 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.509459 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/824cbd14-4a34-4df1-864e-63bb34900aab-credential-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:08.509579 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.509476 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/824cbd14-4a34-4df1-864e-63bb34900aab-istiod-ca-cert\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:08.509579 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.509492 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/824cbd14-4a34-4df1-864e-63bb34900aab-istio-data\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:08.509579 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.509505 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/824cbd14-4a34-4df1-864e-63bb34900aab-workload-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:08.511375 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.511347 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824cbd14-4a34-4df1-864e-63bb34900aab-istio-envoy" pod "824cbd14-4a34-4df1-864e-63bb34900aab" (UID: "824cbd14-4a34-4df1-864e-63bb34900aab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:38:08.511494 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.511472 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/824cbd14-4a34-4df1-864e-63bb34900aab-kube-api-access-wfp4z" pod "824cbd14-4a34-4df1-864e-63bb34900aab" (UID: "824cbd14-4a34-4df1-864e-63bb34900aab"). InnerVolumeSpecName "kube-api-access-wfp4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:38:08.511534 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.511513 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/824cbd14-4a34-4df1-864e-63bb34900aab-istio-token" pod "824cbd14-4a34-4df1-864e-63bb34900aab" (UID: "824cbd14-4a34-4df1-864e-63bb34900aab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:38:08.511845 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.511824 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/824cbd14-4a34-4df1-864e-63bb34900aab-istio-podinfo" pod "824cbd14-4a34-4df1-864e-63bb34900aab" (UID: "824cbd14-4a34-4df1-864e-63bb34900aab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:38:08.610528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.610497 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/824cbd14-4a34-4df1-864e-63bb34900aab-istio-podinfo\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:08.610528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.610526 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/824cbd14-4a34-4df1-864e-63bb34900aab-istio-token\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:08.610528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.610535 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/824cbd14-4a34-4df1-864e-63bb34900aab-istio-envoy\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:08.610822 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.610545 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfp4z\" (UniqueName: \"kubernetes.io/projected/824cbd14-4a34-4df1-864e-63bb34900aab-kube-api-access-wfp4z\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:08.738139 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.738094 4952 generic.go:356] "Generic (PLEG): container finished" podID="0646012c-55f4-4e2b-8dc8-dcd9603504a3" containerID="d0c3fb9c24656cb620d1a370bb4b5a63286fcd56136944b52989b158225c6d40" exitCode=0 Oct 30 23:38:08.738329 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.738158 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" Oct 30 23:38:08.738403 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.738322 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" event={"ID":"0646012c-55f4-4e2b-8dc8-dcd9603504a3","Type":"ContainerDied","Data":"d0c3fb9c24656cb620d1a370bb4b5a63286fcd56136944b52989b158225c6d40"} Oct 30 23:38:08.738467 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.738411 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz" event={"ID":"0646012c-55f4-4e2b-8dc8-dcd9603504a3","Type":"ContainerDied","Data":"d77b0ea2a9ca13d08649fe225bcaa7895b61a1864e8720d45865b6a2c6d2b6a5"} Oct 30 23:38:08.738467 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.738441 4952 scope.go:122] "RemoveContainer" containerID="d0c3fb9c24656cb620d1a370bb4b5a63286fcd56136944b52989b158225c6d40" Oct 30 23:38:08.740211 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.740187 4952 generic.go:356] "Generic (PLEG): container finished" podID="824cbd14-4a34-4df1-864e-63bb34900aab" containerID="18ad8f0ed81d0e16352354e96f03c6c0defa7d402c7c11102f4f0aaa88ee4b5d" exitCode=0 Oct 30 23:38:08.740324 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.740224 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" event={"ID":"824cbd14-4a34-4df1-864e-63bb34900aab","Type":"ContainerDied","Data":"18ad8f0ed81d0e16352354e96f03c6c0defa7d402c7c11102f4f0aaa88ee4b5d"} Oct 30 23:38:08.740324 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.740253 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" Oct 30 23:38:08.740324 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.740282 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg" event={"ID":"824cbd14-4a34-4df1-864e-63bb34900aab","Type":"ContainerDied","Data":"db357e5d14434ea6504233e41bdc89ed176da3ed50a699bfce98349a9d5e41fa"} Oct 30 23:38:08.742191 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.742166 4952 generic.go:356] "Generic (PLEG): container finished" podID="293e802c-8e13-4e98-9026-8f13ee239f58" containerID="c0d6e0408408a3d44f7780b7f4ed0083b45b700ed3b191b16ce20395de32ca13" exitCode=0 Oct 30 23:38:08.742336 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.742207 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" Oct 30 23:38:08.742336 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.742235 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" event={"ID":"293e802c-8e13-4e98-9026-8f13ee239f58","Type":"ContainerDied","Data":"c0d6e0408408a3d44f7780b7f4ed0083b45b700ed3b191b16ce20395de32ca13"} Oct 30 23:38:08.742336 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.742273 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq" event={"ID":"293e802c-8e13-4e98-9026-8f13ee239f58","Type":"ContainerDied","Data":"645fc1133f50bed09dc6453407def5cbbd62aa3c4e990b440969f90a011dad73"} Oct 30 23:38:08.752615 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.752593 4952 scope.go:122] "RemoveContainer" containerID="d0c3fb9c24656cb620d1a370bb4b5a63286fcd56136944b52989b158225c6d40" Oct 30 23:38:08.753109 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:38:08.752959 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0c3fb9c24656cb620d1a370bb4b5a63286fcd56136944b52989b158225c6d40\": container with ID starting with d0c3fb9c24656cb620d1a370bb4b5a63286fcd56136944b52989b158225c6d40 not found: ID does not exist" containerID="d0c3fb9c24656cb620d1a370bb4b5a63286fcd56136944b52989b158225c6d40" Oct 30 23:38:08.753109 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.752997 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0c3fb9c24656cb620d1a370bb4b5a63286fcd56136944b52989b158225c6d40"} err="failed to get container status \"d0c3fb9c24656cb620d1a370bb4b5a63286fcd56136944b52989b158225c6d40\": rpc error: code = NotFound desc = could not find container \"d0c3fb9c24656cb620d1a370bb4b5a63286fcd56136944b52989b158225c6d40\": container with ID starting with d0c3fb9c24656cb620d1a370bb4b5a63286fcd56136944b52989b158225c6d40 not found: ID does not exist" Oct 30 23:38:08.753109 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.753020 4952 scope.go:122] "RemoveContainer" containerID="18ad8f0ed81d0e16352354e96f03c6c0defa7d402c7c11102f4f0aaa88ee4b5d" Oct 30 23:38:08.766183 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.765007 4952 scope.go:122] "RemoveContainer" containerID="18ad8f0ed81d0e16352354e96f03c6c0defa7d402c7c11102f4f0aaa88ee4b5d" Oct 30 23:38:08.766821 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:38:08.766796 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18ad8f0ed81d0e16352354e96f03c6c0defa7d402c7c11102f4f0aaa88ee4b5d\": container with ID starting with 18ad8f0ed81d0e16352354e96f03c6c0defa7d402c7c11102f4f0aaa88ee4b5d not found: ID does not exist" containerID="18ad8f0ed81d0e16352354e96f03c6c0defa7d402c7c11102f4f0aaa88ee4b5d" Oct 30 23:38:08.766922 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.766828 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18ad8f0ed81d0e16352354e96f03c6c0defa7d402c7c11102f4f0aaa88ee4b5d"} err="failed to get container status \"18ad8f0ed81d0e16352354e96f03c6c0defa7d402c7c11102f4f0aaa88ee4b5d\": rpc error: code = NotFound desc = could not find container \"18ad8f0ed81d0e16352354e96f03c6c0defa7d402c7c11102f4f0aaa88ee4b5d\": container with ID starting with 18ad8f0ed81d0e16352354e96f03c6c0defa7d402c7c11102f4f0aaa88ee4b5d not found: ID does not exist" Oct 30 23:38:08.766922 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.766848 4952 scope.go:122] "RemoveContainer" containerID="c0d6e0408408a3d44f7780b7f4ed0083b45b700ed3b191b16ce20395de32ca13" Oct 30 23:38:08.780203 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.780181 4952 scope.go:122] "RemoveContainer" containerID="c0d6e0408408a3d44f7780b7f4ed0083b45b700ed3b191b16ce20395de32ca13" Oct 30 23:38:08.780520 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:38:08.780498 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0d6e0408408a3d44f7780b7f4ed0083b45b700ed3b191b16ce20395de32ca13\": container with ID starting with c0d6e0408408a3d44f7780b7f4ed0083b45b700ed3b191b16ce20395de32ca13 not found: ID does not exist" containerID="c0d6e0408408a3d44f7780b7f4ed0083b45b700ed3b191b16ce20395de32ca13" Oct 30 23:38:08.780601 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.780533 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0d6e0408408a3d44f7780b7f4ed0083b45b700ed3b191b16ce20395de32ca13"} err="failed to get container status \"c0d6e0408408a3d44f7780b7f4ed0083b45b700ed3b191b16ce20395de32ca13\": rpc error: code = NotFound desc = could not find container \"c0d6e0408408a3d44f7780b7f4ed0083b45b700ed3b191b16ce20395de32ca13\": container with ID starting with c0d6e0408408a3d44f7780b7f4ed0083b45b700ed3b191b16ce20395de32ca13 not found: ID does not exist" Oct 30 23:38:08.904938 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.904898 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz"] Oct 30 23:38:08.908681 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.908661 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-g6vzz"] Oct 30 23:38:08.973017 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:08.972977 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0646012c-55f4-4e2b-8dc8-dcd9603504a3" path="/var/lib/kubelet/pods/0646012c-55f4-4e2b-8dc8-dcd9603504a3/volumes" Oct 30 23:38:09.016696 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.016625 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg"] Oct 30 23:38:09.021815 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.021791 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zn2pg"] Oct 30 23:38:09.138806 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.138760 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq"] Oct 30 23:38:09.154839 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.154807 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-8whnq"] Oct 30 23:38:09.463383 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.463360 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" Oct 30 23:38:09.523744 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.523712 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/482b6308-b55b-445f-b1a4-f95bab24f844-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/482b6308-b55b-445f-b1a4-f95bab24f844-credential-socket\") pod \"482b6308-b55b-445f-b1a4-f95bab24f844\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " Oct 30 23:38:09.524231 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.523791 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/482b6308-b55b-445f-b1a4-f95bab24f844-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/482b6308-b55b-445f-b1a4-f95bab24f844-istiod-ca-cert\") pod \"482b6308-b55b-445f-b1a4-f95bab24f844\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " Oct 30 23:38:09.524231 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.523809 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/482b6308-b55b-445f-b1a4-f95bab24f844-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/482b6308-b55b-445f-b1a4-f95bab24f844-istio-data\") pod \"482b6308-b55b-445f-b1a4-f95bab24f844\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " Oct 30 23:38:09.524231 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.523827 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/482b6308-b55b-445f-b1a4-f95bab24f844-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/482b6308-b55b-445f-b1a4-f95bab24f844-istio-envoy\") pod \"482b6308-b55b-445f-b1a4-f95bab24f844\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " Oct 30 23:38:09.524231 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.523867 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/482b6308-b55b-445f-b1a4-f95bab24f844-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/482b6308-b55b-445f-b1a4-f95bab24f844-workload-certs\") pod \"482b6308-b55b-445f-b1a4-f95bab24f844\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " Oct 30 23:38:09.524231 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.523916 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/482b6308-b55b-445f-b1a4-f95bab24f844-kube-api-access-8rprf\" (UniqueName: \"kubernetes.io/projected/482b6308-b55b-445f-b1a4-f95bab24f844-kube-api-access-8rprf\") pod \"482b6308-b55b-445f-b1a4-f95bab24f844\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " Oct 30 23:38:09.524231 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.523989 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/482b6308-b55b-445f-b1a4-f95bab24f844-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/482b6308-b55b-445f-b1a4-f95bab24f844-workload-socket\") pod \"482b6308-b55b-445f-b1a4-f95bab24f844\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " Oct 30 23:38:09.524231 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.523990 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/482b6308-b55b-445f-b1a4-f95bab24f844-credential-socket" pod "482b6308-b55b-445f-b1a4-f95bab24f844" (UID: "482b6308-b55b-445f-b1a4-f95bab24f844"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:38:09.524231 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.524139 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/482b6308-b55b-445f-b1a4-f95bab24f844-istio-token\" (UniqueName: \"kubernetes.io/projected/482b6308-b55b-445f-b1a4-f95bab24f844-istio-token\") pod \"482b6308-b55b-445f-b1a4-f95bab24f844\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " Oct 30 23:38:09.524649 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.524255 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/482b6308-b55b-445f-b1a4-f95bab24f844-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/482b6308-b55b-445f-b1a4-f95bab24f844-istio-podinfo\") pod \"482b6308-b55b-445f-b1a4-f95bab24f844\" (UID: \"482b6308-b55b-445f-b1a4-f95bab24f844\") " Oct 30 23:38:09.524649 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.524262 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/482b6308-b55b-445f-b1a4-f95bab24f844-istio-data" pod "482b6308-b55b-445f-b1a4-f95bab24f844" (UID: "482b6308-b55b-445f-b1a4-f95bab24f844"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:38:09.524649 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.524283 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/482b6308-b55b-445f-b1a4-f95bab24f844-workload-certs" pod "482b6308-b55b-445f-b1a4-f95bab24f844" (UID: "482b6308-b55b-445f-b1a4-f95bab24f844"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:38:09.524649 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.524329 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/482b6308-b55b-445f-b1a4-f95bab24f844-istiod-ca-cert" pod "482b6308-b55b-445f-b1a4-f95bab24f844" (UID: "482b6308-b55b-445f-b1a4-f95bab24f844"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:38:09.524649 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.524601 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/482b6308-b55b-445f-b1a4-f95bab24f844-workload-socket" pod "482b6308-b55b-445f-b1a4-f95bab24f844" (UID: "482b6308-b55b-445f-b1a4-f95bab24f844"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:38:09.525118 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.524809 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/482b6308-b55b-445f-b1a4-f95bab24f844-istiod-ca-cert\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:09.525118 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.524833 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/482b6308-b55b-445f-b1a4-f95bab24f844-istio-data\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:09.525118 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.524847 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/482b6308-b55b-445f-b1a4-f95bab24f844-workload-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:09.525118 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.524860 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/482b6308-b55b-445f-b1a4-f95bab24f844-workload-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:09.525118 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.524872 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/482b6308-b55b-445f-b1a4-f95bab24f844-credential-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:09.526602 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.526574 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/482b6308-b55b-445f-b1a4-f95bab24f844-kube-api-access-8rprf" pod "482b6308-b55b-445f-b1a4-f95bab24f844" (UID: "482b6308-b55b-445f-b1a4-f95bab24f844"). InnerVolumeSpecName "kube-api-access-8rprf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:38:09.526789 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.526765 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/482b6308-b55b-445f-b1a4-f95bab24f844-istio-envoy" pod "482b6308-b55b-445f-b1a4-f95bab24f844" (UID: "482b6308-b55b-445f-b1a4-f95bab24f844"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:38:09.526881 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.526859 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/482b6308-b55b-445f-b1a4-f95bab24f844-istio-token" pod "482b6308-b55b-445f-b1a4-f95bab24f844" (UID: "482b6308-b55b-445f-b1a4-f95bab24f844"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:38:09.527553 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.527534 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/482b6308-b55b-445f-b1a4-f95bab24f844-istio-podinfo" pod "482b6308-b55b-445f-b1a4-f95bab24f844" (UID: "482b6308-b55b-445f-b1a4-f95bab24f844"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:38:09.620291 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.620256 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:38:09.626660 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.626632 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rprf\" (UniqueName: \"kubernetes.io/projected/482b6308-b55b-445f-b1a4-f95bab24f844-kube-api-access-8rprf\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:09.626660 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.626660 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/482b6308-b55b-445f-b1a4-f95bab24f844-istio-token\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:09.626828 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.626670 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/482b6308-b55b-445f-b1a4-f95bab24f844-istio-podinfo\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:09.626828 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.626679 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/482b6308-b55b-445f-b1a4-f95bab24f844-istio-envoy\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:38:09.751370 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.751340 4952 generic.go:356] "Generic (PLEG): container finished" podID="482b6308-b55b-445f-b1a4-f95bab24f844" containerID="f09fd34c21cdd17cb1aa3ad23dc79b4840838bc9b358c9566c7fd598ffdabeee" exitCode=0 Oct 30 23:38:09.751574 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.751388 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" Oct 30 23:38:09.751574 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.751419 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" event={"ID":"482b6308-b55b-445f-b1a4-f95bab24f844","Type":"ContainerDied","Data":"f09fd34c21cdd17cb1aa3ad23dc79b4840838bc9b358c9566c7fd598ffdabeee"} Oct 30 23:38:09.751574 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.751470 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj" event={"ID":"482b6308-b55b-445f-b1a4-f95bab24f844","Type":"ContainerDied","Data":"36d4e8b797b22674f75705979d2164e07ecc870b30bfc5a0a287a2c86c071f2f"} Oct 30 23:38:09.751574 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.751500 4952 scope.go:122] "RemoveContainer" containerID="f09fd34c21cdd17cb1aa3ad23dc79b4840838bc9b358c9566c7fd598ffdabeee" Oct 30 23:38:09.763724 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.763703 4952 scope.go:122] "RemoveContainer" containerID="f09fd34c21cdd17cb1aa3ad23dc79b4840838bc9b358c9566c7fd598ffdabeee" Oct 30 23:38:09.763986 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:38:09.763965 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f09fd34c21cdd17cb1aa3ad23dc79b4840838bc9b358c9566c7fd598ffdabeee\": container with ID starting with f09fd34c21cdd17cb1aa3ad23dc79b4840838bc9b358c9566c7fd598ffdabeee not found: ID does not exist" containerID="f09fd34c21cdd17cb1aa3ad23dc79b4840838bc9b358c9566c7fd598ffdabeee" Oct 30 23:38:09.764148 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.763997 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f09fd34c21cdd17cb1aa3ad23dc79b4840838bc9b358c9566c7fd598ffdabeee"} err="failed to get container status \"f09fd34c21cdd17cb1aa3ad23dc79b4840838bc9b358c9566c7fd598ffdabeee\": rpc error: code = NotFound desc = could not find container \"f09fd34c21cdd17cb1aa3ad23dc79b4840838bc9b358c9566c7fd598ffdabeee\": container with ID starting with f09fd34c21cdd17cb1aa3ad23dc79b4840838bc9b358c9566c7fd598ffdabeee not found: ID does not exist" Oct 30 23:38:09.918481 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.918445 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj"] Oct 30 23:38:09.922165 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:09.922142 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-4ntpj"] Oct 30 23:38:10.972767 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:10.972737 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="293e802c-8e13-4e98-9026-8f13ee239f58" path="/var/lib/kubelet/pods/293e802c-8e13-4e98-9026-8f13ee239f58/volumes" Oct 30 23:38:10.973248 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:10.973232 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="482b6308-b55b-445f-b1a4-f95bab24f844" path="/var/lib/kubelet/pods/482b6308-b55b-445f-b1a4-f95bab24f844/volumes" Oct 30 23:38:10.973685 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:10.973672 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="824cbd14-4a34-4df1-864e-63bb34900aab" path="/var/lib/kubelet/pods/824cbd14-4a34-4df1-864e-63bb34900aab/volumes" Oct 30 23:38:13.750850 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:13.750772 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:38:14.810014 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:14.809983 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:38:16.723249 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:16.723203 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:38:16.947488 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:16.947449 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:38:19.617146 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:19.617116 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk"] Oct 30 23:38:19.618097 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:19.618075 4952 cpu_manager.go:414] "containerMap: removing container" podUID="c0e935ae-8758-4f00-beca-d6b4c7c38dd6" containerName="istio-proxy" Oct 30 23:38:19.618208 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:19.618100 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0e935ae-8758-4f00-beca-d6b4c7c38dd6" containerName="istio-proxy" Oct 30 23:38:19.618208 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:19.618119 4952 cpu_manager.go:414] "containerMap: removing container" podUID="0646012c-55f4-4e2b-8dc8-dcd9603504a3" containerName="istio-proxy" Oct 30 23:38:19.618208 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:19.618127 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0646012c-55f4-4e2b-8dc8-dcd9603504a3" containerName="istio-proxy" Oct 30 23:38:19.618208 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:19.618148 4952 cpu_manager.go:414] "containerMap: removing container" podUID="824cbd14-4a34-4df1-864e-63bb34900aab" containerName="istio-proxy" Oct 30 23:38:19.618208 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:19.618157 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="824cbd14-4a34-4df1-864e-63bb34900aab" containerName="istio-proxy" Oct 30 23:38:19.618208 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:19.618199 4952 cpu_manager.go:414] "containerMap: removing container" podUID="293e802c-8e13-4e98-9026-8f13ee239f58" containerName="istio-proxy" Oct 30 23:38:19.618208 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:19.618206 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="293e802c-8e13-4e98-9026-8f13ee239f58" containerName="istio-proxy" Oct 30 23:38:19.618434 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:19.618218 4952 cpu_manager.go:414] "containerMap: removing container" podUID="482b6308-b55b-445f-b1a4-f95bab24f844" containerName="istio-proxy" Oct 30 23:38:19.618434 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:19.618226 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="482b6308-b55b-445f-b1a4-f95bab24f844" containerName="istio-proxy" Oct 30 23:38:19.618434 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:19.618375 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="293e802c-8e13-4e98-9026-8f13ee239f58" containerName="istio-proxy" Oct 30 23:38:19.618434 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:19.618388 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="482b6308-b55b-445f-b1a4-f95bab24f844" containerName="istio-proxy" Oct 30 23:38:19.618434 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:19.618407 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0e935ae-8758-4f00-beca-d6b4c7c38dd6" containerName="istio-proxy" Oct 30 23:38:19.618434 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:19.618420 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="0646012c-55f4-4e2b-8dc8-dcd9603504a3" containerName="istio-proxy" Oct 30 23:38:19.618434 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:19.618432 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="824cbd14-4a34-4df1-864e-63bb34900aab" containerName="istio-proxy" Oct 30 23:38:19.622312 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:19.622291 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk" Oct 30 23:38:19.724314 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:19.724278 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-ba13e7-serving-cert\"" Oct 30 23:38:19.728642 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:19.728615 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk"] Oct 30 23:38:19.791028 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:19.790986 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b155a013-f16a-481e-a4f0-a1afd8140a9f-maas-api-tls\") pod \"maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk\" (UID: \"b155a013-f16a-481e-a4f0-a1afd8140a9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk" Oct 30 23:38:19.791246 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:19.791148 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b155a013-f16a-481e-a4f0-a1afd8140a9f-metrics-certs\") pod \"maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk\" (UID: \"b155a013-f16a-481e-a4f0-a1afd8140a9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk" Oct 30 23:38:19.791246 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:19.791205 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnvwj\" (UniqueName: \"kubernetes.io/projected/b155a013-f16a-481e-a4f0-a1afd8140a9f-kube-api-access-qnvwj\") pod \"maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk\" (UID: \"b155a013-f16a-481e-a4f0-a1afd8140a9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk" Oct 30 23:38:19.791518 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:19.791491 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b155a013-f16a-481e-a4f0-a1afd8140a9f-openshift-service-ca\") pod \"maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk\" (UID: \"b155a013-f16a-481e-a4f0-a1afd8140a9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk" Oct 30 23:38:19.892605 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:19.892530 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnvwj\" (UniqueName: \"kubernetes.io/projected/b155a013-f16a-481e-a4f0-a1afd8140a9f-kube-api-access-qnvwj\") pod \"maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk\" (UID: \"b155a013-f16a-481e-a4f0-a1afd8140a9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk" Oct 30 23:38:19.892751 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:19.892723 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b155a013-f16a-481e-a4f0-a1afd8140a9f-openshift-service-ca\") pod \"maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk\" (UID: \"b155a013-f16a-481e-a4f0-a1afd8140a9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk" Oct 30 23:38:19.892794 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:19.892765 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b155a013-f16a-481e-a4f0-a1afd8140a9f-maas-api-tls\") pod \"maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk\" (UID: \"b155a013-f16a-481e-a4f0-a1afd8140a9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk" Oct 30 23:38:19.892848 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:19.892795 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b155a013-f16a-481e-a4f0-a1afd8140a9f-metrics-certs\") pod \"maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk\" (UID: \"b155a013-f16a-481e-a4f0-a1afd8140a9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk" Oct 30 23:38:19.892926 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:38:19.892908 4952 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-ba13e7-serving-cert: secret "maas-api-e2e-api-b-ba13e7-serving-cert" not found Oct 30 23:38:19.893001 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:38:19.892990 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b155a013-f16a-481e-a4f0-a1afd8140a9f-maas-api-tls podName:b155a013-f16a-481e-a4f0-a1afd8140a9f nodeName:}" failed. No retries permitted until 2026-10-30 23:38:20.392969411 +0000 UTC m=+1919.536799608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b155a013-f16a-481e-a4f0-a1afd8140a9f-maas-api-tls") pod "maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk" (UID: "b155a013-f16a-481e-a4f0-a1afd8140a9f") : secret "maas-api-e2e-api-b-ba13e7-serving-cert" not found Oct 30 23:38:19.893429 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:19.893407 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b155a013-f16a-481e-a4f0-a1afd8140a9f-openshift-service-ca\") pod \"maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk\" (UID: \"b155a013-f16a-481e-a4f0-a1afd8140a9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk" Oct 30 23:38:19.895590 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:19.895562 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b155a013-f16a-481e-a4f0-a1afd8140a9f-metrics-certs\") pod \"maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk\" (UID: \"b155a013-f16a-481e-a4f0-a1afd8140a9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk" Oct 30 23:38:19.952797 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:19.952763 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnvwj\" (UniqueName: \"kubernetes.io/projected/b155a013-f16a-481e-a4f0-a1afd8140a9f-kube-api-access-qnvwj\") pod \"maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk\" (UID: \"b155a013-f16a-481e-a4f0-a1afd8140a9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk" Oct 30 23:38:19.957911 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:19.957868 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:38:20.195571 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:20.195472 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:38:20.399746 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:20.399705 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b155a013-f16a-481e-a4f0-a1afd8140a9f-maas-api-tls\") pod \"maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk\" (UID: \"b155a013-f16a-481e-a4f0-a1afd8140a9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk" Oct 30 23:38:20.402527 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:20.402499 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b155a013-f16a-481e-a4f0-a1afd8140a9f-maas-api-tls\") pod \"maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk\" (UID: \"b155a013-f16a-481e-a4f0-a1afd8140a9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk" Oct 30 23:38:20.422612 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:20.422579 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:38:20.538369 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:20.538285 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk" Oct 30 23:38:20.935194 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:20.935162 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk"] Oct 30 23:38:20.935700 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:38:20.935669 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb155a013_f16a_481e_a4f0_a1afd8140a9f.slice/crio-26e8d7bb1dc501e4fe358066cd317effcb71e9f7a25aa7996d2a51596111eeee WatchSource:0}: Error finding container 26e8d7bb1dc501e4fe358066cd317effcb71e9f7a25aa7996d2a51596111eeee: Status 404 returned error can't find the container with id 26e8d7bb1dc501e4fe358066cd317effcb71e9f7a25aa7996d2a51596111eeee Oct 30 23:38:20.937424 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:20.937398 4952 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 23:38:21.153375 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:38:21.153339 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:38:21.414915 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:21.414880 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:38:21.905497 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:21.905456 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk" event={"ID":"b155a013-f16a-481e-a4f0-a1afd8140a9f","Type":"ContainerStarted","Data":"26e8d7bb1dc501e4fe358066cd317effcb71e9f7a25aa7996d2a51596111eeee"} Oct 30 23:38:23.045068 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:23.044404 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk" event={"ID":"b155a013-f16a-481e-a4f0-a1afd8140a9f","Type":"ContainerStarted","Data":"61f01fc60b51dc83ecd71ed87e7267e659cd877a8c225c3d4e18eb74b6052764"} Oct 30 23:38:23.047989 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:23.046349 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk" Oct 30 23:38:23.164968 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:23.164916 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk" podStartSLOduration=3.173110794 podStartE2EDuration="4.164903656s" totalImagesPullingTime="991.792862ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:38:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:38:23.161132191 +0000 UTC m=+1922.304962417" watchObservedRunningTime="2026-10-30 23:38:23.164903656 +0000 UTC m=+1922.308733862" Oct 30 23:38:25.223317 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:25.223255 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:38:29.633308 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:29.633275 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bcc619-775dbf957-vpn64"] Oct 30 23:38:29.637684 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:29.637658 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcc619-775dbf957-vpn64" Oct 30 23:38:29.739489 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:29.739453 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f85606f-6cb6-442e-a5e1-3a57fb732f93-openshift-service-ca\") pod \"maas-api-e2e-api-a-bcc619-775dbf957-vpn64\" (UID: \"4f85606f-6cb6-442e-a5e1-3a57fb732f93\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcc619-775dbf957-vpn64" Oct 30 23:38:29.739642 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:29.739528 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f85606f-6cb6-442e-a5e1-3a57fb732f93-metrics-certs\") pod \"maas-api-e2e-api-a-bcc619-775dbf957-vpn64\" (UID: \"4f85606f-6cb6-442e-a5e1-3a57fb732f93\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcc619-775dbf957-vpn64" Oct 30 23:38:29.739708 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:29.739648 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f85606f-6cb6-442e-a5e1-3a57fb732f93-maas-api-tls\") pod \"maas-api-e2e-api-a-bcc619-775dbf957-vpn64\" (UID: \"4f85606f-6cb6-442e-a5e1-3a57fb732f93\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcc619-775dbf957-vpn64" Oct 30 23:38:29.739759 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:29.739751 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7qjk\" (UniqueName: \"kubernetes.io/projected/4f85606f-6cb6-442e-a5e1-3a57fb732f93-kube-api-access-t7qjk\") pod \"maas-api-e2e-api-a-bcc619-775dbf957-vpn64\" (UID: \"4f85606f-6cb6-442e-a5e1-3a57fb732f93\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcc619-775dbf957-vpn64" Oct 30 23:38:29.742358 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:29.742339 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-bcc619-serving-cert\"" Oct 30 23:38:29.746366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:29.746346 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bcc619-775dbf957-vpn64"] Oct 30 23:38:29.841576 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:29.841546 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f85606f-6cb6-442e-a5e1-3a57fb732f93-openshift-service-ca\") pod \"maas-api-e2e-api-a-bcc619-775dbf957-vpn64\" (UID: \"4f85606f-6cb6-442e-a5e1-3a57fb732f93\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcc619-775dbf957-vpn64" Oct 30 23:38:29.841747 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:29.841634 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f85606f-6cb6-442e-a5e1-3a57fb732f93-metrics-certs\") pod \"maas-api-e2e-api-a-bcc619-775dbf957-vpn64\" (UID: \"4f85606f-6cb6-442e-a5e1-3a57fb732f93\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcc619-775dbf957-vpn64" Oct 30 23:38:29.841747 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:29.841678 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f85606f-6cb6-442e-a5e1-3a57fb732f93-maas-api-tls\") pod \"maas-api-e2e-api-a-bcc619-775dbf957-vpn64\" (UID: \"4f85606f-6cb6-442e-a5e1-3a57fb732f93\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcc619-775dbf957-vpn64" Oct 30 23:38:29.841747 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:29.841726 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7qjk\" (UniqueName: \"kubernetes.io/projected/4f85606f-6cb6-442e-a5e1-3a57fb732f93-kube-api-access-t7qjk\") pod \"maas-api-e2e-api-a-bcc619-775dbf957-vpn64\" (UID: \"4f85606f-6cb6-442e-a5e1-3a57fb732f93\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcc619-775dbf957-vpn64" Oct 30 23:38:29.841919 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:38:29.841841 4952 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-bcc619-serving-cert: secret "maas-api-e2e-api-a-bcc619-serving-cert" not found Oct 30 23:38:29.841919 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:38:29.841909 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f85606f-6cb6-442e-a5e1-3a57fb732f93-maas-api-tls podName:4f85606f-6cb6-442e-a5e1-3a57fb732f93 nodeName:}" failed. No retries permitted until 2026-10-30 23:38:30.341890364 +0000 UTC m=+1929.485720550 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4f85606f-6cb6-442e-a5e1-3a57fb732f93-maas-api-tls") pod "maas-api-e2e-api-a-bcc619-775dbf957-vpn64" (UID: "4f85606f-6cb6-442e-a5e1-3a57fb732f93") : secret "maas-api-e2e-api-a-bcc619-serving-cert" not found Oct 30 23:38:29.842246 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:29.842224 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f85606f-6cb6-442e-a5e1-3a57fb732f93-openshift-service-ca\") pod \"maas-api-e2e-api-a-bcc619-775dbf957-vpn64\" (UID: \"4f85606f-6cb6-442e-a5e1-3a57fb732f93\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcc619-775dbf957-vpn64" Oct 30 23:38:29.844463 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:29.844442 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f85606f-6cb6-442e-a5e1-3a57fb732f93-metrics-certs\") pod \"maas-api-e2e-api-a-bcc619-775dbf957-vpn64\" (UID: \"4f85606f-6cb6-442e-a5e1-3a57fb732f93\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcc619-775dbf957-vpn64" Oct 30 23:38:29.902380 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:29.902300 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7qjk\" (UniqueName: \"kubernetes.io/projected/4f85606f-6cb6-442e-a5e1-3a57fb732f93-kube-api-access-t7qjk\") pod \"maas-api-e2e-api-a-bcc619-775dbf957-vpn64\" (UID: \"4f85606f-6cb6-442e-a5e1-3a57fb732f93\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcc619-775dbf957-vpn64" Oct 30 23:38:29.920485 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:29.920406 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:38:30.063850 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:30.063822 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk" Oct 30 23:38:30.151525 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:30.151469 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:38:30.348776 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:30.348746 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f85606f-6cb6-442e-a5e1-3a57fb732f93-maas-api-tls\") pod \"maas-api-e2e-api-a-bcc619-775dbf957-vpn64\" (UID: \"4f85606f-6cb6-442e-a5e1-3a57fb732f93\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcc619-775dbf957-vpn64" Oct 30 23:38:30.351434 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:30.351415 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f85606f-6cb6-442e-a5e1-3a57fb732f93-maas-api-tls\") pod \"maas-api-e2e-api-a-bcc619-775dbf957-vpn64\" (UID: \"4f85606f-6cb6-442e-a5e1-3a57fb732f93\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcc619-775dbf957-vpn64" Oct 30 23:38:30.374486 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:30.374460 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:38:30.551561 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:30.551536 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcc619-775dbf957-vpn64" Oct 30 23:38:30.938967 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:38:30.938936 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f85606f_6cb6_442e_a5e1_3a57fb732f93.slice/crio-34589a869082fed513fe45e7d7d69c62853eb651504d73011d6a33f62622b05d WatchSource:0}: Error finding container 34589a869082fed513fe45e7d7d69c62853eb651504d73011d6a33f62622b05d: Status 404 returned error can't find the container with id 34589a869082fed513fe45e7d7d69c62853eb651504d73011d6a33f62622b05d Oct 30 23:38:30.939604 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:30.939572 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bcc619-775dbf957-vpn64"] Oct 30 23:38:31.101089 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:31.101060 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcc619-775dbf957-vpn64" event={"ID":"4f85606f-6cb6-442e-a5e1-3a57fb732f93","Type":"ContainerStarted","Data":"34589a869082fed513fe45e7d7d69c62853eb651504d73011d6a33f62622b05d"} Oct 30 23:38:32.110596 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:32.110561 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcc619-775dbf957-vpn64" event={"ID":"4f85606f-6cb6-442e-a5e1-3a57fb732f93","Type":"ContainerStarted","Data":"820c25471f94e0f5786d795da0e58cf0c11194b5b3a5c40d321e0b34d571e6d6"} Oct 30 23:38:32.110964 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:32.110651 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcc619-775dbf957-vpn64" Oct 30 23:38:32.227366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:32.227320 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcc619-775dbf957-vpn64" podStartSLOduration=2.353627386 podStartE2EDuration="3.227304151s" totalImagesPullingTime="873.676765ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:38:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:38:32.22620346 +0000 UTC m=+1931.370033664" watchObservedRunningTime="2026-10-30 23:38:32.227304151 +0000 UTC m=+1931.371134357" Oct 30 23:38:34.882210 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:34.882167 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:38:38.125244 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:38.125214 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcc619-775dbf957-vpn64" Oct 30 23:38:39.672844 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:39.672804 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:38:44.468319 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:44.468275 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:38:49.110157 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:49.110119 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:38:49.341431 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:49.341392 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:38:51.142491 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:38:51.142456 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:38:54.087180 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:54.087149 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:38:58.436088 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:58.436056 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8"] Oct 30 23:38:58.440393 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:58.440376 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8" Oct 30 23:38:58.492771 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:58.492747 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b526ac-serving-cert\"" Oct 30 23:38:58.537235 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:58.537201 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/164790dc-301c-40a2-a62a-665146b9adc2-maas-api-tls\") pod \"maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8\" (UID: \"164790dc-301c-40a2-a62a-665146b9adc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8" Oct 30 23:38:58.537400 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:58.537245 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhkh5\" (UniqueName: \"kubernetes.io/projected/164790dc-301c-40a2-a62a-665146b9adc2-kube-api-access-bhkh5\") pod \"maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8\" (UID: \"164790dc-301c-40a2-a62a-665146b9adc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8" Oct 30 23:38:58.537479 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:58.537436 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/164790dc-301c-40a2-a62a-665146b9adc2-metrics-certs\") pod \"maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8\" (UID: \"164790dc-301c-40a2-a62a-665146b9adc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8" Oct 30 23:38:58.537639 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:58.537601 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/164790dc-301c-40a2-a62a-665146b9adc2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8\" (UID: \"164790dc-301c-40a2-a62a-665146b9adc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8" Oct 30 23:38:58.546472 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:58.546446 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8"] Oct 30 23:38:58.638401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:58.638372 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/164790dc-301c-40a2-a62a-665146b9adc2-metrics-certs\") pod \"maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8\" (UID: \"164790dc-301c-40a2-a62a-665146b9adc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8" Oct 30 23:38:58.638577 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:58.638461 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/164790dc-301c-40a2-a62a-665146b9adc2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8\" (UID: \"164790dc-301c-40a2-a62a-665146b9adc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8" Oct 30 23:38:58.638577 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:58.638527 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/164790dc-301c-40a2-a62a-665146b9adc2-maas-api-tls\") pod \"maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8\" (UID: \"164790dc-301c-40a2-a62a-665146b9adc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8" Oct 30 23:38:58.638577 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:58.638545 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhkh5\" (UniqueName: \"kubernetes.io/projected/164790dc-301c-40a2-a62a-665146b9adc2-kube-api-access-bhkh5\") pod \"maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8\" (UID: \"164790dc-301c-40a2-a62a-665146b9adc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8" Oct 30 23:38:58.638760 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:38:58.638635 4952 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-b526ac-serving-cert: secret "maas-api-e2e-inf-a-b526ac-serving-cert" not found Oct 30 23:38:58.638760 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:38:58.638698 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/164790dc-301c-40a2-a62a-665146b9adc2-maas-api-tls podName:164790dc-301c-40a2-a62a-665146b9adc2 nodeName:}" failed. No retries permitted until 2026-10-30 23:38:59.138683681 +0000 UTC m=+1958.282513893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/164790dc-301c-40a2-a62a-665146b9adc2-maas-api-tls") pod "maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8" (UID: "164790dc-301c-40a2-a62a-665146b9adc2") : secret "maas-api-e2e-inf-a-b526ac-serving-cert" not found Oct 30 23:38:58.639242 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:58.639185 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/164790dc-301c-40a2-a62a-665146b9adc2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8\" (UID: \"164790dc-301c-40a2-a62a-665146b9adc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8" Oct 30 23:38:58.640894 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:58.640877 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/164790dc-301c-40a2-a62a-665146b9adc2-metrics-certs\") pod \"maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8\" (UID: \"164790dc-301c-40a2-a62a-665146b9adc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8" Oct 30 23:38:58.696291 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:58.696223 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhkh5\" (UniqueName: \"kubernetes.io/projected/164790dc-301c-40a2-a62a-665146b9adc2-kube-api-access-bhkh5\") pod \"maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8\" (UID: \"164790dc-301c-40a2-a62a-665146b9adc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8" Oct 30 23:38:58.728800 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:58.728761 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:38:58.952945 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:58.952865 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:38:59.146716 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:59.146684 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/164790dc-301c-40a2-a62a-665146b9adc2-maas-api-tls\") pod \"maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8\" (UID: \"164790dc-301c-40a2-a62a-665146b9adc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8" Oct 30 23:38:59.149305 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:59.149278 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/164790dc-301c-40a2-a62a-665146b9adc2-maas-api-tls\") pod \"maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8\" (UID: \"164790dc-301c-40a2-a62a-665146b9adc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8" Oct 30 23:38:59.354894 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:59.354863 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8" Oct 30 23:38:59.748068 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:38:59.748019 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8"] Oct 30 23:38:59.748635 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:38:59.748612 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod164790dc_301c_40a2_a62a_665146b9adc2.slice/crio-9c22f3778252c03f23598a2975c9d14d4e1cf0ff21a96a89c545beb07a2adb57 WatchSource:0}: Error finding container 9c22f3778252c03f23598a2975c9d14d4e1cf0ff21a96a89c545beb07a2adb57: Status 404 returned error can't find the container with id 9c22f3778252c03f23598a2975c9d14d4e1cf0ff21a96a89c545beb07a2adb57 Oct 30 23:39:00.337890 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:00.337849 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8" event={"ID":"164790dc-301c-40a2-a62a-665146b9adc2","Type":"ContainerStarted","Data":"9c22f3778252c03f23598a2975c9d14d4e1cf0ff21a96a89c545beb07a2adb57"} Oct 30 23:39:01.346902 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:01.346869 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8" event={"ID":"164790dc-301c-40a2-a62a-665146b9adc2","Type":"ContainerStarted","Data":"8f4ce9805e58c1343efc67f5ee2441faffa72471c612af1e3c3901036025f4c0"} Oct 30 23:39:01.347337 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:01.346932 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8" Oct 30 23:39:01.486118 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:01.486062 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8" podStartSLOduration=2.5627052 podStartE2EDuration="3.486028127s" totalImagesPullingTime="923.322927ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:38:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:39:01.485378289 +0000 UTC m=+1960.629208522" watchObservedRunningTime="2026-10-30 23:39:01.486028127 +0000 UTC m=+1960.629858333" Oct 30 23:39:03.908407 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:03.908359 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:39:07.358662 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:07.358632 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8" Oct 30 23:39:08.706019 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:08.705987 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:39:13.538460 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:13.538408 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:39:18.267765 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:18.267722 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:39:21.142968 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:39:21.142933 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:39:21.763113 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:21.763073 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:39:22.934611 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:22.934579 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:39:23.155660 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:23.155630 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:39:23.501520 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:23.501470 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:39:27.981276 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:27.981240 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:39:31.761067 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:31.760999 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:39:31.988506 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:31.988477 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:39:33.305675 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:33.305635 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:39:33.914946 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:33.914901 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:39:37.874697 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:37.874630 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:39:38.113297 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:38.113257 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:39:38.530160 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:38.530109 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:39:39.098307 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:39.098267 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:39:42.825219 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:42.825185 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:39:43.281205 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:43.281161 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:39:43.605454 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:43.605422 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:39:43.836330 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:43.836294 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:39:44.068366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:44.068292 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:39:44.606549 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:44.606512 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:39:44.897436 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:44.897337 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:39:48.065394 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:48.065358 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:39:48.302209 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:48.302173 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:39:48.745583 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:48.745537 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:39:49.551286 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:49.551253 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:39:49.813810 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:49.813725 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:39:51.147076 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:39:51.147016 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:39:52.974985 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:52.974952 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:39:53.204460 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:53.204413 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:39:54.156649 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:54.156612 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:39:59.247608 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:59.247549 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:39:59.466122 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:39:59.466087 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:04.376065 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:04.375959 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr"] Oct 30 23:40:04.376562 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:04.376214 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" podUID="1ca32053-b157-4808-b0e8-c7a2c02a58b9" containerName="istio-proxy" containerID="cri-o://edcf8dfcafe5bed2ae29dd5e87cdf6854338b0a1fe273ac492a037affb170921" gracePeriod=30 Oct 30 23:40:04.832127 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:04.832013 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:05.295390 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:05.295350 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287"] Oct 30 23:40:05.295604 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:05.295581 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" podUID="2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0" containerName="istio-proxy" containerID="cri-o://9d2acd35e934df2a372acbb25ca4f11629e3ab8af05df898cfd5fdd4993ded71" gracePeriod=30 Oct 30 23:40:05.749153 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:05.749115 4952 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 23:40:05.749653 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:05.749183 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" podUID="1ca32053-b157-4808-b0e8-c7a2c02a58b9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:40:06.022733 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:06.022661 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:06.790715 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:06.790687 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:07.471112 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:07.470979 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:07.927718 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:07.927688 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 23:40:07.928123 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:07.927737 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" podUID="2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:40:08.104090 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:08.104024 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:09.683123 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:09.683098 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" Oct 30 23:40:09.765551 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:09.765519 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ca32053-b157-4808-b0e8-c7a2c02a58b9-istio-token\" (UniqueName: \"kubernetes.io/projected/1ca32053-b157-4808-b0e8-c7a2c02a58b9-istio-token\") pod \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " Oct 30 23:40:09.765699 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:09.765623 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ca32053-b157-4808-b0e8-c7a2c02a58b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ca32053-b157-4808-b0e8-c7a2c02a58b9-istio-data\") pod \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " Oct 30 23:40:09.765766 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:09.765726 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ca32053-b157-4808-b0e8-c7a2c02a58b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca32053-b157-4808-b0e8-c7a2c02a58b9-workload-socket\") pod \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " Oct 30 23:40:09.765825 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:09.765774 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ca32053-b157-4808-b0e8-c7a2c02a58b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ca32053-b157-4808-b0e8-c7a2c02a58b9-istio-envoy\") pod \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " Oct 30 23:40:09.765825 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:09.765811 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ca32053-b157-4808-b0e8-c7a2c02a58b9-kube-api-access-7dkzc\" (UniqueName: \"kubernetes.io/projected/1ca32053-b157-4808-b0e8-c7a2c02a58b9-kube-api-access-7dkzc\") pod \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " Oct 30 23:40:09.765929 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:09.765866 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ca32053-b157-4808-b0e8-c7a2c02a58b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ca32053-b157-4808-b0e8-c7a2c02a58b9-istiod-ca-cert\") pod \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " Oct 30 23:40:09.765929 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:09.765908 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ca32053-b157-4808-b0e8-c7a2c02a58b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ca32053-b157-4808-b0e8-c7a2c02a58b9-istio-podinfo\") pod \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " Oct 30 23:40:09.766060 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:09.765918 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca32053-b157-4808-b0e8-c7a2c02a58b9-istio-data" pod "1ca32053-b157-4808-b0e8-c7a2c02a58b9" (UID: "1ca32053-b157-4808-b0e8-c7a2c02a58b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:40:09.766060 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:09.765998 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca32053-b157-4808-b0e8-c7a2c02a58b9-workload-socket" pod "1ca32053-b157-4808-b0e8-c7a2c02a58b9" (UID: "1ca32053-b157-4808-b0e8-c7a2c02a58b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:40:09.766184 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:09.766097 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ca32053-b157-4808-b0e8-c7a2c02a58b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca32053-b157-4808-b0e8-c7a2c02a58b9-credential-socket\") pod \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " Oct 30 23:40:09.766184 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:09.766142 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ca32053-b157-4808-b0e8-c7a2c02a58b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ca32053-b157-4808-b0e8-c7a2c02a58b9-workload-certs\") pod \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\" (UID: \"1ca32053-b157-4808-b0e8-c7a2c02a58b9\") " Oct 30 23:40:09.766615 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:09.766547 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ca32053-b157-4808-b0e8-c7a2c02a58b9-istiod-ca-cert" pod "1ca32053-b157-4808-b0e8-c7a2c02a58b9" (UID: "1ca32053-b157-4808-b0e8-c7a2c02a58b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:40:09.766615 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:09.766596 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca32053-b157-4808-b0e8-c7a2c02a58b9-credential-socket" pod "1ca32053-b157-4808-b0e8-c7a2c02a58b9" (UID: "1ca32053-b157-4808-b0e8-c7a2c02a58b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:40:09.766932 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:09.766908 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca32053-b157-4808-b0e8-c7a2c02a58b9-workload-certs" pod "1ca32053-b157-4808-b0e8-c7a2c02a58b9" (UID: "1ca32053-b157-4808-b0e8-c7a2c02a58b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:40:09.766999 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:09.766937 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ca32053-b157-4808-b0e8-c7a2c02a58b9-istiod-ca-cert\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:09.766999 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:09.766957 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca32053-b157-4808-b0e8-c7a2c02a58b9-credential-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:09.766999 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:09.766971 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ca32053-b157-4808-b0e8-c7a2c02a58b9-istio-data\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:09.766999 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:09.766984 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca32053-b157-4808-b0e8-c7a2c02a58b9-workload-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:09.768149 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:09.768120 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca32053-b157-4808-b0e8-c7a2c02a58b9-istio-envoy" pod "1ca32053-b157-4808-b0e8-c7a2c02a58b9" (UID: "1ca32053-b157-4808-b0e8-c7a2c02a58b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:40:09.768257 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:09.768159 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca32053-b157-4808-b0e8-c7a2c02a58b9-istio-token" pod "1ca32053-b157-4808-b0e8-c7a2c02a58b9" (UID: "1ca32053-b157-4808-b0e8-c7a2c02a58b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:40:09.768257 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:09.768166 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca32053-b157-4808-b0e8-c7a2c02a58b9-kube-api-access-7dkzc" pod "1ca32053-b157-4808-b0e8-c7a2c02a58b9" (UID: "1ca32053-b157-4808-b0e8-c7a2c02a58b9"). InnerVolumeSpecName "kube-api-access-7dkzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:40:09.768337 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:09.768271 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ca32053-b157-4808-b0e8-c7a2c02a58b9-istio-podinfo" pod "1ca32053-b157-4808-b0e8-c7a2c02a58b9" (UID: "1ca32053-b157-4808-b0e8-c7a2c02a58b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:40:09.868922 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:09.868886 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ca32053-b157-4808-b0e8-c7a2c02a58b9-istio-envoy\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:09.868922 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:09.868925 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dkzc\" (UniqueName: \"kubernetes.io/projected/1ca32053-b157-4808-b0e8-c7a2c02a58b9-kube-api-access-7dkzc\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:09.869153 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:09.868941 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ca32053-b157-4808-b0e8-c7a2c02a58b9-istio-podinfo\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:09.869153 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:09.868956 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ca32053-b157-4808-b0e8-c7a2c02a58b9-workload-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:09.869153 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:09.868972 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ca32053-b157-4808-b0e8-c7a2c02a58b9-istio-token\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:09.935714 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:09.935639 4952 generic.go:356] "Generic (PLEG): container finished" podID="1ca32053-b157-4808-b0e8-c7a2c02a58b9" containerID="edcf8dfcafe5bed2ae29dd5e87cdf6854338b0a1fe273ac492a037affb170921" exitCode=0 Oct 30 23:40:09.935714 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:09.935690 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" Oct 30 23:40:09.935903 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:09.935725 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" event={"ID":"1ca32053-b157-4808-b0e8-c7a2c02a58b9","Type":"ContainerDied","Data":"edcf8dfcafe5bed2ae29dd5e87cdf6854338b0a1fe273ac492a037affb170921"} Oct 30 23:40:09.935903 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:09.935785 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr" event={"ID":"1ca32053-b157-4808-b0e8-c7a2c02a58b9","Type":"ContainerDied","Data":"78dd3fde84bee9a72f474c9a9d6c1e512c108c9c05b20a793f8284e01da110d0"} Oct 30 23:40:09.935903 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:09.935807 4952 scope.go:122] "RemoveContainer" containerID="edcf8dfcafe5bed2ae29dd5e87cdf6854338b0a1fe273ac492a037affb170921" Oct 30 23:40:09.950252 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:09.950229 4952 scope.go:122] "RemoveContainer" containerID="edcf8dfcafe5bed2ae29dd5e87cdf6854338b0a1fe273ac492a037affb170921" Oct 30 23:40:09.950641 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:40:09.950619 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edcf8dfcafe5bed2ae29dd5e87cdf6854338b0a1fe273ac492a037affb170921\": container with ID starting with edcf8dfcafe5bed2ae29dd5e87cdf6854338b0a1fe273ac492a037affb170921 not found: ID does not exist" containerID="edcf8dfcafe5bed2ae29dd5e87cdf6854338b0a1fe273ac492a037affb170921" Oct 30 23:40:09.950701 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:09.950649 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edcf8dfcafe5bed2ae29dd5e87cdf6854338b0a1fe273ac492a037affb170921"} err="failed to get container status \"edcf8dfcafe5bed2ae29dd5e87cdf6854338b0a1fe273ac492a037affb170921\": rpc error: code = NotFound desc = could not find container \"edcf8dfcafe5bed2ae29dd5e87cdf6854338b0a1fe273ac492a037affb170921\": container with ID starting with edcf8dfcafe5bed2ae29dd5e87cdf6854338b0a1fe273ac492a037affb170921 not found: ID does not exist" Oct 30 23:40:10.104927 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.104861 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr"] Oct 30 23:40:10.109690 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.109661 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-f2qnr"] Oct 30 23:40:10.134290 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.134256 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:10.570683 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.570623 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" Oct 30 23:40:10.685868 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.685838 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-workload-certs\") pod \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " Oct 30 23:40:10.686344 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.685926 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-workload-socket\") pod \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " Oct 30 23:40:10.686344 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.685948 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-kube-api-access-qdfdl\" (UniqueName: \"kubernetes.io/projected/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-kube-api-access-qdfdl\") pod \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " Oct 30 23:40:10.686344 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.685986 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-istio-token\" (UniqueName: \"kubernetes.io/projected/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-istio-token\") pod \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " Oct 30 23:40:10.686344 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.686011 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-istio-envoy\") pod \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " Oct 30 23:40:10.686344 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.686081 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-credential-socket\") pod \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " Oct 30 23:40:10.686344 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.686164 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-istiod-ca-cert\") pod \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " Oct 30 23:40:10.686344 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.686169 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-workload-certs" pod "2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0" (UID: "2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:40:10.686344 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.686223 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-istio-data\") pod \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " Oct 30 23:40:10.686344 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.686302 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-workload-socket" pod "2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0" (UID: "2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:40:10.686344 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.686321 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-istio-podinfo\") pod \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\" (UID: \"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0\") " Oct 30 23:40:10.687077 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.686542 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-istio-data" pod "2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0" (UID: "2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:40:10.687077 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.686585 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-istiod-ca-cert" pod "2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0" (UID: "2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:40:10.687077 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.686713 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-credential-socket" pod "2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0" (UID: "2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:40:10.687077 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.686913 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-workload-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:10.687077 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.686929 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-credential-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:10.687077 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.686939 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-istiod-ca-cert\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:10.687077 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.686951 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-istio-data\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:10.687077 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.686964 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-workload-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:10.689352 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.689324 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-istio-envoy" pod "2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0" (UID: "2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:40:10.689352 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.688822 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-istio-podinfo" pod "2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0" (UID: "2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:40:10.690653 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.690629 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-kube-api-access-qdfdl" pod "2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0" (UID: "2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0"). InnerVolumeSpecName "kube-api-access-qdfdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:40:10.690782 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.690658 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-istio-token" pod "2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0" (UID: "2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:40:10.789031 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.788998 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdfdl\" (UniqueName: \"kubernetes.io/projected/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-kube-api-access-qdfdl\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:10.789197 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.789084 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-istio-token\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:10.789197 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.789110 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-istio-envoy\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:10.789197 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.789123 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0-istio-podinfo\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:10.945588 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.945538 4952 generic.go:356] "Generic (PLEG): container finished" podID="2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0" containerID="9d2acd35e934df2a372acbb25ca4f11629e3ab8af05df898cfd5fdd4993ded71" exitCode=0 Oct 30 23:40:10.945798 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.945604 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" Oct 30 23:40:10.945798 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.945620 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" event={"ID":"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0","Type":"ContainerDied","Data":"9d2acd35e934df2a372acbb25ca4f11629e3ab8af05df898cfd5fdd4993ded71"} Oct 30 23:40:10.945798 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.945687 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287" event={"ID":"2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0","Type":"ContainerDied","Data":"c9e4adb5e77e0688186812781eeec9d8098bf299f7ddaf2b897066cc3be80eaa"} Oct 30 23:40:10.945798 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.945713 4952 scope.go:122] "RemoveContainer" containerID="9d2acd35e934df2a372acbb25ca4f11629e3ab8af05df898cfd5fdd4993ded71" Oct 30 23:40:10.957588 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.957573 4952 scope.go:122] "RemoveContainer" containerID="9d2acd35e934df2a372acbb25ca4f11629e3ab8af05df898cfd5fdd4993ded71" Oct 30 23:40:10.957835 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:40:10.957815 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d2acd35e934df2a372acbb25ca4f11629e3ab8af05df898cfd5fdd4993ded71\": container with ID starting with 9d2acd35e934df2a372acbb25ca4f11629e3ab8af05df898cfd5fdd4993ded71 not found: ID does not exist" containerID="9d2acd35e934df2a372acbb25ca4f11629e3ab8af05df898cfd5fdd4993ded71" Oct 30 23:40:10.957908 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.957843 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d2acd35e934df2a372acbb25ca4f11629e3ab8af05df898cfd5fdd4993ded71"} err="failed to get container status \"9d2acd35e934df2a372acbb25ca4f11629e3ab8af05df898cfd5fdd4993ded71\": rpc error: code = NotFound desc = could not find container \"9d2acd35e934df2a372acbb25ca4f11629e3ab8af05df898cfd5fdd4993ded71\": container with ID starting with 9d2acd35e934df2a372acbb25ca4f11629e3ab8af05df898cfd5fdd4993ded71 not found: ID does not exist" Oct 30 23:40:10.971115 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:10.971094 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ca32053-b157-4808-b0e8-c7a2c02a58b9" path="/var/lib/kubelet/pods/1ca32053-b157-4808-b0e8-c7a2c02a58b9/volumes" Oct 30 23:40:11.119800 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:11.119773 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287"] Oct 30 23:40:11.169081 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:11.169052 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-wd287"] Oct 30 23:40:12.078347 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:12.078315 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:12.343062 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:12.341863 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:12.973488 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:12.973457 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0" path="/var/lib/kubelet/pods/2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0/volumes" Oct 30 23:40:15.313100 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:15.313061 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:17.682988 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:17.682957 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:19.812699 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:19.812664 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6"] Oct 30 23:40:19.813121 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:19.812856 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" podUID="7ac8fce7-3841-4fb4-a7a3-06359477953b" containerName="istio-proxy" containerID="cri-o://f948ee65804062388c8493c7835ae278365527341d2d6f6d9f50f67c5936ba10" gracePeriod=30 Oct 30 23:40:19.854763 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:19.854736 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7"] Oct 30 23:40:19.854955 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:19.854922 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" podUID="3f222119-3b8c-4f8a-8ba2-88ef4d873da4" containerName="istio-proxy" containerID="cri-o://9a1e81f25be020d55e9bfe7661664bb4e1d5bbc2489d4fba85dc0bf6d6d82736" gracePeriod=30 Oct 30 23:40:20.239999 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:20.239962 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:20.428968 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:20.428855 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889"] Oct 30 23:40:20.429832 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:20.429143 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" podUID="3ad4e638-0d0d-423b-96db-f671e8095f08" containerName="istio-proxy" containerID="cri-o://6006616f914193bef622bbf1bb6966adf82da6c3fa10853b936b8ba9104d69e7" gracePeriod=30 Oct 30 23:40:20.466910 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:20.466876 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:20.688721 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:20.688683 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-26ce6b-6964c5f8b7-xmfd7"] Oct 30 23:40:20.689377 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:20.689360 4952 cpu_manager.go:414] "containerMap: removing container" podUID="1ca32053-b157-4808-b0e8-c7a2c02a58b9" containerName="istio-proxy" Oct 30 23:40:20.689377 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:20.689378 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ca32053-b157-4808-b0e8-c7a2c02a58b9" containerName="istio-proxy" Oct 30 23:40:20.689556 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:20.689410 4952 cpu_manager.go:414] "containerMap: removing container" podUID="2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0" containerName="istio-proxy" Oct 30 23:40:20.689556 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:20.689415 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0" containerName="istio-proxy" Oct 30 23:40:20.689556 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:20.689518 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ecaaa2d-a6ac-4efc-ac36-559fa792d3b0" containerName="istio-proxy" Oct 30 23:40:20.689556 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:20.689534 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ca32053-b157-4808-b0e8-c7a2c02a58b9" containerName="istio-proxy" Oct 30 23:40:20.693315 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:20.693294 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26ce6b-6964c5f8b7-xmfd7" Oct 30 23:40:20.699846 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:20.699820 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:20.745541 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:20.745520 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-26ce6b\"" Oct 30 23:40:20.745541 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:20.745526 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-26ce6b-dockercfg-swjlf\"" Oct 30 23:40:20.749985 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:20.749965 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-26ce6b-6964c5f8b7-xmfd7"] Oct 30 23:40:20.871102 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:20.854849 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxp49\" (UniqueName: \"kubernetes.io/projected/3b443f22-9bac-4a46-9b4a-8982e0006fc4-kube-api-access-mxp49\") pod \"payload-pre-processing-e2e-api-b-26ce6b-6964c5f8b7-xmfd7\" (UID: \"3b443f22-9bac-4a46-9b4a-8982e0006fc4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-26ce6b-6964c5f8b7-xmfd7" Oct 30 23:40:20.873631 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:20.873601 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b443f22-9bac-4a46-9b4a-8982e0006fc4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-26ce6b-6964c5f8b7-xmfd7\" (UID: \"3b443f22-9bac-4a46-9b4a-8982e0006fc4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-26ce6b-6964c5f8b7-xmfd7" Oct 30 23:40:20.975705 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:20.975625 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxp49\" (UniqueName: \"kubernetes.io/projected/3b443f22-9bac-4a46-9b4a-8982e0006fc4-kube-api-access-mxp49\") pod \"payload-pre-processing-e2e-api-b-26ce6b-6964c5f8b7-xmfd7\" (UID: \"3b443f22-9bac-4a46-9b4a-8982e0006fc4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-26ce6b-6964c5f8b7-xmfd7" Oct 30 23:40:20.975947 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:20.975848 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b443f22-9bac-4a46-9b4a-8982e0006fc4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-26ce6b-6964c5f8b7-xmfd7\" (UID: \"3b443f22-9bac-4a46-9b4a-8982e0006fc4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-26ce6b-6964c5f8b7-xmfd7" Oct 30 23:40:21.026653 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:21.026623 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-26ce6b\"" Oct 30 23:40:21.032628 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:21.032604 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxp49\" (UniqueName: \"kubernetes.io/projected/3b443f22-9bac-4a46-9b4a-8982e0006fc4-kube-api-access-mxp49\") pod \"payload-pre-processing-e2e-api-b-26ce6b-6964c5f8b7-xmfd7\" (UID: \"3b443f22-9bac-4a46-9b4a-8982e0006fc4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-26ce6b-6964c5f8b7-xmfd7" Oct 30 23:40:21.038643 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:21.038607 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b443f22-9bac-4a46-9b4a-8982e0006fc4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-26ce6b-6964c5f8b7-xmfd7\" (UID: \"3b443f22-9bac-4a46-9b4a-8982e0006fc4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-26ce6b-6964c5f8b7-xmfd7" Oct 30 23:40:21.147561 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:40:21.147527 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:40:21.181311 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:21.181269 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 23:40:21.181486 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:21.181342 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" podUID="7ac8fce7-3841-4fb4-a7a3-06359477953b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:40:21.360769 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:21.360742 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-26ce6b-dockercfg-swjlf\"" Oct 30 23:40:21.370864 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:21.370836 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26ce6b-6964c5f8b7-xmfd7" Oct 30 23:40:21.808654 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:21.808620 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-26ce6b-6964c5f8b7-xmfd7"] Oct 30 23:40:21.809645 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:40:21.809614 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b443f22_9bac_4a46_9b4a_8982e0006fc4.slice/crio-368fb23d3b83bc40a729ef401e95e5199b6dba3e0409631f3e4f535686978f4e WatchSource:0}: Error finding container 368fb23d3b83bc40a729ef401e95e5199b6dba3e0409631f3e4f535686978f4e: Status 404 returned error can't find the container with id 368fb23d3b83bc40a729ef401e95e5199b6dba3e0409631f3e4f535686978f4e Oct 30 23:40:22.027646 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:22.027613 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26ce6b-6964c5f8b7-xmfd7" event={"ID":"3b443f22-9bac-4a46-9b4a-8982e0006fc4","Type":"ContainerStarted","Data":"93aed4ffe1c2fb328cef0b48b3c6dfe2306fb367535f63d33d854904f630b027"} Oct 30 23:40:22.028018 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:22.027664 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26ce6b-6964c5f8b7-xmfd7" event={"ID":"3b443f22-9bac-4a46-9b4a-8982e0006fc4","Type":"ContainerStarted","Data":"368fb23d3b83bc40a729ef401e95e5199b6dba3e0409631f3e4f535686978f4e"} Oct 30 23:40:22.028018 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:22.027710 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26ce6b-6964c5f8b7-xmfd7" Oct 30 23:40:22.104883 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:22.104858 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:22.141595 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:22.141555 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26ce6b-6964c5f8b7-xmfd7" podStartSLOduration=2.141544273 podStartE2EDuration="2.141544273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:22.140627837 +0000 UTC m=+2041.284458068" watchObservedRunningTime="2026-10-30 23:40:22.141544273 +0000 UTC m=+2041.285374540" Oct 30 23:40:22.341232 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:22.341191 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:23.238001 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:23.237968 4952 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 23:40:23.238442 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:23.238048 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" podUID="3f222119-3b8c-4f8a-8ba2-88ef4d873da4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:40:23.620801 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:23.620764 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 23:40:23.621025 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:23.620823 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" podUID="3ad4e638-0d0d-423b-96db-f671e8095f08" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:40:25.089743 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.089710 4952 generic.go:356] "Generic (PLEG): container finished" podID="7ac8fce7-3841-4fb4-a7a3-06359477953b" containerID="f948ee65804062388c8493c7835ae278365527341d2d6f6d9f50f67c5936ba10" exitCode=0 Oct 30 23:40:25.090255 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.089749 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" event={"ID":"7ac8fce7-3841-4fb4-a7a3-06359477953b","Type":"ContainerDied","Data":"f948ee65804062388c8493c7835ae278365527341d2d6f6d9f50f67c5936ba10"} Oct 30 23:40:25.092687 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.092662 4952 generic.go:356] "Generic (PLEG): container finished" podID="3f222119-3b8c-4f8a-8ba2-88ef4d873da4" containerID="9a1e81f25be020d55e9bfe7661664bb4e1d5bbc2489d4fba85dc0bf6d6d82736" exitCode=0 Oct 30 23:40:25.092804 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.092690 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" event={"ID":"3f222119-3b8c-4f8a-8ba2-88ef4d873da4","Type":"ContainerDied","Data":"9a1e81f25be020d55e9bfe7661664bb4e1d5bbc2489d4fba85dc0bf6d6d82736"} Oct 30 23:40:25.101273 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.101249 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:25.157814 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.157783 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" Oct 30 23:40:25.235601 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.235579 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" Oct 30 23:40:25.239687 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.239664 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ac8fce7-3841-4fb4-a7a3-06359477953b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ac8fce7-3841-4fb4-a7a3-06359477953b-istiod-ca-cert\") pod \"7ac8fce7-3841-4fb4-a7a3-06359477953b\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " Oct 30 23:40:25.239789 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.239736 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ac8fce7-3841-4fb4-a7a3-06359477953b-istio-token\" (UniqueName: \"kubernetes.io/projected/7ac8fce7-3841-4fb4-a7a3-06359477953b-istio-token\") pod \"7ac8fce7-3841-4fb4-a7a3-06359477953b\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " Oct 30 23:40:25.239860 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.239837 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ac8fce7-3841-4fb4-a7a3-06359477953b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ac8fce7-3841-4fb4-a7a3-06359477953b-istio-data\") pod \"7ac8fce7-3841-4fb4-a7a3-06359477953b\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " Oct 30 23:40:25.239917 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.239890 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ac8fce7-3841-4fb4-a7a3-06359477953b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac8fce7-3841-4fb4-a7a3-06359477953b-workload-socket\") pod \"7ac8fce7-3841-4fb4-a7a3-06359477953b\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " Oct 30 23:40:25.239969 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.239951 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ac8fce7-3841-4fb4-a7a3-06359477953b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac8fce7-3841-4fb4-a7a3-06359477953b-credential-socket\") pod \"7ac8fce7-3841-4fb4-a7a3-06359477953b\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " Oct 30 23:40:25.240021 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.239972 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ac8fce7-3841-4fb4-a7a3-06359477953b-istiod-ca-cert" pod "7ac8fce7-3841-4fb4-a7a3-06359477953b" (UID: "7ac8fce7-3841-4fb4-a7a3-06359477953b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:40:25.240021 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.239982 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ac8fce7-3841-4fb4-a7a3-06359477953b-kube-api-access-blndn\" (UniqueName: \"kubernetes.io/projected/7ac8fce7-3841-4fb4-a7a3-06359477953b-kube-api-access-blndn\") pod \"7ac8fce7-3841-4fb4-a7a3-06359477953b\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " Oct 30 23:40:25.240182 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.240165 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ac8fce7-3841-4fb4-a7a3-06359477953b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ac8fce7-3841-4fb4-a7a3-06359477953b-istio-envoy\") pod \"7ac8fce7-3841-4fb4-a7a3-06359477953b\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " Oct 30 23:40:25.240246 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.240198 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ac8fce7-3841-4fb4-a7a3-06359477953b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ac8fce7-3841-4fb4-a7a3-06359477953b-istio-podinfo\") pod \"7ac8fce7-3841-4fb4-a7a3-06359477953b\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " Oct 30 23:40:25.240305 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.240148 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac8fce7-3841-4fb4-a7a3-06359477953b-istio-data" pod "7ac8fce7-3841-4fb4-a7a3-06359477953b" (UID: "7ac8fce7-3841-4fb4-a7a3-06359477953b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:40:25.240305 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.240255 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ac8fce7-3841-4fb4-a7a3-06359477953b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ac8fce7-3841-4fb4-a7a3-06359477953b-workload-certs\") pod \"7ac8fce7-3841-4fb4-a7a3-06359477953b\" (UID: \"7ac8fce7-3841-4fb4-a7a3-06359477953b\") " Oct 30 23:40:25.240305 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.240255 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac8fce7-3841-4fb4-a7a3-06359477953b-workload-socket" pod "7ac8fce7-3841-4fb4-a7a3-06359477953b" (UID: "7ac8fce7-3841-4fb4-a7a3-06359477953b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:40:25.240305 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.240285 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac8fce7-3841-4fb4-a7a3-06359477953b-credential-socket" pod "7ac8fce7-3841-4fb4-a7a3-06359477953b" (UID: "7ac8fce7-3841-4fb4-a7a3-06359477953b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:40:25.240659 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.240637 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac8fce7-3841-4fb4-a7a3-06359477953b-workload-certs" pod "7ac8fce7-3841-4fb4-a7a3-06359477953b" (UID: "7ac8fce7-3841-4fb4-a7a3-06359477953b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:40:25.240980 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.240957 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ac8fce7-3841-4fb4-a7a3-06359477953b-istio-data\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:25.241059 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.240990 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac8fce7-3841-4fb4-a7a3-06359477953b-workload-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:25.241059 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.241007 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ac8fce7-3841-4fb4-a7a3-06359477953b-credential-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:25.241059 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.241024 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ac8fce7-3841-4fb4-a7a3-06359477953b-workload-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:25.241187 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.241061 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ac8fce7-3841-4fb4-a7a3-06359477953b-istiod-ca-cert\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:25.242151 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.242130 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ac8fce7-3841-4fb4-a7a3-06359477953b-istio-token" pod "7ac8fce7-3841-4fb4-a7a3-06359477953b" (UID: "7ac8fce7-3841-4fb4-a7a3-06359477953b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:40:25.242385 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.242363 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ac8fce7-3841-4fb4-a7a3-06359477953b-istio-podinfo" pod "7ac8fce7-3841-4fb4-a7a3-06359477953b" (UID: "7ac8fce7-3841-4fb4-a7a3-06359477953b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:40:25.242706 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.242686 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ac8fce7-3841-4fb4-a7a3-06359477953b-kube-api-access-blndn" pod "7ac8fce7-3841-4fb4-a7a3-06359477953b" (UID: "7ac8fce7-3841-4fb4-a7a3-06359477953b"). InnerVolumeSpecName "kube-api-access-blndn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:40:25.242781 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.242687 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac8fce7-3841-4fb4-a7a3-06359477953b-istio-envoy" pod "7ac8fce7-3841-4fb4-a7a3-06359477953b" (UID: "7ac8fce7-3841-4fb4-a7a3-06359477953b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:40:25.342620 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.342502 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-istio-podinfo\") pod \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " Oct 30 23:40:25.342620 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.342562 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-istio-envoy\") pod \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " Oct 30 23:40:25.342620 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.342595 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-kube-api-access-4l4j8\" (UniqueName: \"kubernetes.io/projected/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-kube-api-access-4l4j8\") pod \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " Oct 30 23:40:25.342921 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.342712 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-workload-certs\") pod \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " Oct 30 23:40:25.342921 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.342811 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-istiod-ca-cert\") pod \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " Oct 30 23:40:25.342921 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.342844 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-istio-token\" (UniqueName: \"kubernetes.io/projected/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-istio-token\") pod \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " Oct 30 23:40:25.342921 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.342888 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-credential-socket\") pod \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " Oct 30 23:40:25.342921 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.342918 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-workload-socket\") pod \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " Oct 30 23:40:25.343123 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.342965 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-istio-data\") pod \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\" (UID: \"3f222119-3b8c-4f8a-8ba2-88ef4d873da4\") " Oct 30 23:40:25.343662 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.343641 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ac8fce7-3841-4fb4-a7a3-06359477953b-istio-envoy\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:25.343699 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.343670 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ac8fce7-3841-4fb4-a7a3-06359477953b-istio-podinfo\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:25.343699 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.343686 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ac8fce7-3841-4fb4-a7a3-06359477953b-istio-token\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:25.343764 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.343700 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blndn\" (UniqueName: \"kubernetes.io/projected/7ac8fce7-3841-4fb4-a7a3-06359477953b-kube-api-access-blndn\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:25.344540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.344137 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-workload-certs" pod "3f222119-3b8c-4f8a-8ba2-88ef4d873da4" (UID: "3f222119-3b8c-4f8a-8ba2-88ef4d873da4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:40:25.344540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.344291 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-credential-socket" pod "3f222119-3b8c-4f8a-8ba2-88ef4d873da4" (UID: "3f222119-3b8c-4f8a-8ba2-88ef4d873da4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:40:25.344540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.344505 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-workload-socket" pod "3f222119-3b8c-4f8a-8ba2-88ef4d873da4" (UID: "3f222119-3b8c-4f8a-8ba2-88ef4d873da4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:40:25.344540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.344526 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-istio-data" pod "3f222119-3b8c-4f8a-8ba2-88ef4d873da4" (UID: "3f222119-3b8c-4f8a-8ba2-88ef4d873da4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:40:25.344842 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.344602 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-istiod-ca-cert" pod "3f222119-3b8c-4f8a-8ba2-88ef4d873da4" (UID: "3f222119-3b8c-4f8a-8ba2-88ef4d873da4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:40:25.346718 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.346687 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-istio-token" pod "3f222119-3b8c-4f8a-8ba2-88ef4d873da4" (UID: "3f222119-3b8c-4f8a-8ba2-88ef4d873da4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:40:25.346828 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.346777 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-istio-podinfo" pod "3f222119-3b8c-4f8a-8ba2-88ef4d873da4" (UID: "3f222119-3b8c-4f8a-8ba2-88ef4d873da4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:40:25.346923 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.346903 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-kube-api-access-4l4j8" pod "3f222119-3b8c-4f8a-8ba2-88ef4d873da4" (UID: "3f222119-3b8c-4f8a-8ba2-88ef4d873da4"). InnerVolumeSpecName "kube-api-access-4l4j8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:40:25.347490 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.347466 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-istio-envoy" pod "3f222119-3b8c-4f8a-8ba2-88ef4d873da4" (UID: "3f222119-3b8c-4f8a-8ba2-88ef4d873da4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:40:25.444840 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.444806 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-workload-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:25.444840 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.444836 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-istiod-ca-cert\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:25.444840 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.444845 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-istio-token\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:25.445182 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.444854 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-credential-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:25.445182 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.444863 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-workload-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:25.445182 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.444872 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-istio-data\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:25.445182 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.444879 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-istio-podinfo\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:25.445182 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.444887 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-istio-envoy\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:25.445182 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.444896 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l4j8\" (UniqueName: \"kubernetes.io/projected/3f222119-3b8c-4f8a-8ba2-88ef4d873da4-kube-api-access-4l4j8\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:25.703844 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.703815 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" Oct 30 23:40:25.848955 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.848925 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ad4e638-0d0d-423b-96db-f671e8095f08-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad4e638-0d0d-423b-96db-f671e8095f08-workload-socket\") pod \"3ad4e638-0d0d-423b-96db-f671e8095f08\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " Oct 30 23:40:25.849103 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.848982 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ad4e638-0d0d-423b-96db-f671e8095f08-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad4e638-0d0d-423b-96db-f671e8095f08-istio-podinfo\") pod \"3ad4e638-0d0d-423b-96db-f671e8095f08\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " Oct 30 23:40:25.849103 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.849019 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ad4e638-0d0d-423b-96db-f671e8095f08-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad4e638-0d0d-423b-96db-f671e8095f08-credential-socket\") pod \"3ad4e638-0d0d-423b-96db-f671e8095f08\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " Oct 30 23:40:25.849244 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.849143 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ad4e638-0d0d-423b-96db-f671e8095f08-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad4e638-0d0d-423b-96db-f671e8095f08-workload-certs\") pod \"3ad4e638-0d0d-423b-96db-f671e8095f08\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " Oct 30 23:40:25.849244 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.849203 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ad4e638-0d0d-423b-96db-f671e8095f08-istio-token\" (UniqueName: \"kubernetes.io/projected/3ad4e638-0d0d-423b-96db-f671e8095f08-istio-token\") pod \"3ad4e638-0d0d-423b-96db-f671e8095f08\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " Oct 30 23:40:25.849348 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.849277 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ad4e638-0d0d-423b-96db-f671e8095f08-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad4e638-0d0d-423b-96db-f671e8095f08-istio-data\") pod \"3ad4e638-0d0d-423b-96db-f671e8095f08\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " Oct 30 23:40:25.849400 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.849360 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ad4e638-0d0d-423b-96db-f671e8095f08-kube-api-access-x28dr\" (UniqueName: \"kubernetes.io/projected/3ad4e638-0d0d-423b-96db-f671e8095f08-kube-api-access-x28dr\") pod \"3ad4e638-0d0d-423b-96db-f671e8095f08\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " Oct 30 23:40:25.849400 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.849367 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad4e638-0d0d-423b-96db-f671e8095f08-workload-socket" pod "3ad4e638-0d0d-423b-96db-f671e8095f08" (UID: "3ad4e638-0d0d-423b-96db-f671e8095f08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:40:25.849507 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.849462 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ad4e638-0d0d-423b-96db-f671e8095f08-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad4e638-0d0d-423b-96db-f671e8095f08-istio-envoy\") pod \"3ad4e638-0d0d-423b-96db-f671e8095f08\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " Oct 30 23:40:25.849507 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.849495 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ad4e638-0d0d-423b-96db-f671e8095f08-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad4e638-0d0d-423b-96db-f671e8095f08-istiod-ca-cert\") pod \"3ad4e638-0d0d-423b-96db-f671e8095f08\" (UID: \"3ad4e638-0d0d-423b-96db-f671e8095f08\") " Oct 30 23:40:25.849902 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.849873 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad4e638-0d0d-423b-96db-f671e8095f08-istio-data" pod "3ad4e638-0d0d-423b-96db-f671e8095f08" (UID: "3ad4e638-0d0d-423b-96db-f671e8095f08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:40:25.850211 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.850189 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad4e638-0d0d-423b-96db-f671e8095f08-workload-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:25.850355 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.850218 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad4e638-0d0d-423b-96db-f671e8095f08-istio-data\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:25.850355 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.850240 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad4e638-0d0d-423b-96db-f671e8095f08-credential-socket" pod "3ad4e638-0d0d-423b-96db-f671e8095f08" (UID: "3ad4e638-0d0d-423b-96db-f671e8095f08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:40:25.850355 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.850276 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad4e638-0d0d-423b-96db-f671e8095f08-workload-certs" pod "3ad4e638-0d0d-423b-96db-f671e8095f08" (UID: "3ad4e638-0d0d-423b-96db-f671e8095f08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:40:25.850541 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.850480 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ad4e638-0d0d-423b-96db-f671e8095f08-istiod-ca-cert" pod "3ad4e638-0d0d-423b-96db-f671e8095f08" (UID: "3ad4e638-0d0d-423b-96db-f671e8095f08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:40:25.852731 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.852705 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ad4e638-0d0d-423b-96db-f671e8095f08-istio-podinfo" pod "3ad4e638-0d0d-423b-96db-f671e8095f08" (UID: "3ad4e638-0d0d-423b-96db-f671e8095f08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:40:25.852819 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.852704 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad4e638-0d0d-423b-96db-f671e8095f08-istio-envoy" pod "3ad4e638-0d0d-423b-96db-f671e8095f08" (UID: "3ad4e638-0d0d-423b-96db-f671e8095f08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:40:25.854154 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.854130 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ad4e638-0d0d-423b-96db-f671e8095f08-istio-token" pod "3ad4e638-0d0d-423b-96db-f671e8095f08" (UID: "3ad4e638-0d0d-423b-96db-f671e8095f08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:40:25.854388 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.854311 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ad4e638-0d0d-423b-96db-f671e8095f08-kube-api-access-x28dr" pod "3ad4e638-0d0d-423b-96db-f671e8095f08" (UID: "3ad4e638-0d0d-423b-96db-f671e8095f08"). InnerVolumeSpecName "kube-api-access-x28dr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:40:25.951533 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.951456 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad4e638-0d0d-423b-96db-f671e8095f08-istio-podinfo\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:25.951533 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.951484 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad4e638-0d0d-423b-96db-f671e8095f08-credential-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:25.951533 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.951494 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad4e638-0d0d-423b-96db-f671e8095f08-workload-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:25.951533 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.951504 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ad4e638-0d0d-423b-96db-f671e8095f08-istio-token\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:25.951533 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.951513 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x28dr\" (UniqueName: \"kubernetes.io/projected/3ad4e638-0d0d-423b-96db-f671e8095f08-kube-api-access-x28dr\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:25.951533 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.951522 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad4e638-0d0d-423b-96db-f671e8095f08-istio-envoy\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:25.951533 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:25.951530 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad4e638-0d0d-423b-96db-f671e8095f08-istiod-ca-cert\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:40:26.101518 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:26.101480 4952 generic.go:356] "Generic (PLEG): container finished" podID="3ad4e638-0d0d-423b-96db-f671e8095f08" containerID="6006616f914193bef622bbf1bb6966adf82da6c3fa10853b936b8ba9104d69e7" exitCode=0 Oct 30 23:40:26.101979 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:26.101592 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" Oct 30 23:40:26.101979 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:26.101611 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" event={"ID":"3ad4e638-0d0d-423b-96db-f671e8095f08","Type":"ContainerDied","Data":"6006616f914193bef622bbf1bb6966adf82da6c3fa10853b936b8ba9104d69e7"} Oct 30 23:40:26.101979 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:26.101662 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889" event={"ID":"3ad4e638-0d0d-423b-96db-f671e8095f08","Type":"ContainerDied","Data":"e7c66ef553e45ff8f65f6e9c4a03cda12250821f14e15f213ece30c5a230fd4d"} Oct 30 23:40:26.101979 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:26.101684 4952 scope.go:122] "RemoveContainer" containerID="6006616f914193bef622bbf1bb6966adf82da6c3fa10853b936b8ba9104d69e7" Oct 30 23:40:26.103528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:26.103507 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" Oct 30 23:40:26.103672 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:26.103504 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6" event={"ID":"7ac8fce7-3841-4fb4-a7a3-06359477953b","Type":"ContainerDied","Data":"3f2a0133acf8eae7d8a46cda3d3f75f72f700297e14e38c4270d999f2619070f"} Oct 30 23:40:26.105326 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:26.105292 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" Oct 30 23:40:26.105326 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:26.105318 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7" event={"ID":"3f222119-3b8c-4f8a-8ba2-88ef4d873da4","Type":"ContainerDied","Data":"225c22f8be1ec519a48926eac724d211661cb57c3fa1cbb99fbdbef90ef62738"} Oct 30 23:40:26.116054 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:26.116016 4952 scope.go:122] "RemoveContainer" containerID="6006616f914193bef622bbf1bb6966adf82da6c3fa10853b936b8ba9104d69e7" Oct 30 23:40:26.116388 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:40:26.116369 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6006616f914193bef622bbf1bb6966adf82da6c3fa10853b936b8ba9104d69e7\": container with ID starting with 6006616f914193bef622bbf1bb6966adf82da6c3fa10853b936b8ba9104d69e7 not found: ID does not exist" containerID="6006616f914193bef622bbf1bb6966adf82da6c3fa10853b936b8ba9104d69e7" Oct 30 23:40:26.116443 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:26.116400 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6006616f914193bef622bbf1bb6966adf82da6c3fa10853b936b8ba9104d69e7"} err="failed to get container status \"6006616f914193bef622bbf1bb6966adf82da6c3fa10853b936b8ba9104d69e7\": rpc error: code = NotFound desc = could not find container \"6006616f914193bef622bbf1bb6966adf82da6c3fa10853b936b8ba9104d69e7\": container with ID starting with 6006616f914193bef622bbf1bb6966adf82da6c3fa10853b936b8ba9104d69e7 not found: ID does not exist" Oct 30 23:40:26.116443 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:26.116425 4952 scope.go:122] "RemoveContainer" containerID="f948ee65804062388c8493c7835ae278365527341d2d6f6d9f50f67c5936ba10" Oct 30 23:40:26.127913 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:26.127898 4952 scope.go:122] "RemoveContainer" containerID="9a1e81f25be020d55e9bfe7661664bb4e1d5bbc2489d4fba85dc0bf6d6d82736" Oct 30 23:40:26.270196 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:26.270123 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6"] Oct 30 23:40:26.316953 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:26.316917 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8prc6"] Oct 30 23:40:26.380674 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:26.380642 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7"] Oct 30 23:40:26.394060 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:26.385007 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-dcqg7"] Oct 30 23:40:26.494615 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:26.494591 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889"] Oct 30 23:40:26.498060 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:26.498025 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-28889"] Oct 30 23:40:26.971925 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:26.971890 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ad4e638-0d0d-423b-96db-f671e8095f08" path="/var/lib/kubelet/pods/3ad4e638-0d0d-423b-96db-f671e8095f08/volumes" Oct 30 23:40:26.972436 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:26.972418 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f222119-3b8c-4f8a-8ba2-88ef4d873da4" path="/var/lib/kubelet/pods/3f222119-3b8c-4f8a-8ba2-88ef4d873da4/volumes" Oct 30 23:40:26.972911 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:26.972895 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ac8fce7-3841-4fb4-a7a3-06359477953b" path="/var/lib/kubelet/pods/7ac8fce7-3841-4fb4-a7a3-06359477953b/volumes" Oct 30 23:40:29.884430 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:29.884394 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:30.109402 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:30.109329 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:30.334646 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:30.334560 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:33.039631 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:33.039602 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26ce6b-6964c5f8b7-xmfd7" Oct 30 23:40:34.741627 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:34.741596 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:37.984538 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:37.984502 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:38.204238 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:38.204203 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:39.214504 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:39.214463 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:40.868712 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:40.868674 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:41.107593 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:41.107554 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:41.340279 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:41.340244 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:43.860686 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:43.860651 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:40:45.263452 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:45.263415 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:46.293405 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:46.293374 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:49.384981 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:49.384949 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:49.607435 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:49.607408 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:51.144002 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:40:51.143966 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:40:51.429077 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:51.428913 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:52.728129 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:52.728093 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:52.952775 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:52.952736 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:53.381891 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:53.381856 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-jc5dz"] Oct 30 23:40:53.382703 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:53.382678 4952 cpu_manager.go:414] "containerMap: removing container" podUID="7ac8fce7-3841-4fb4-a7a3-06359477953b" containerName="istio-proxy" Oct 30 23:40:53.382703 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:53.382702 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ac8fce7-3841-4fb4-a7a3-06359477953b" containerName="istio-proxy" Oct 30 23:40:53.382911 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:53.382741 4952 cpu_manager.go:414] "containerMap: removing container" podUID="3ad4e638-0d0d-423b-96db-f671e8095f08" containerName="istio-proxy" Oct 30 23:40:53.382911 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:53.382748 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ad4e638-0d0d-423b-96db-f671e8095f08" containerName="istio-proxy" Oct 30 23:40:53.382911 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:53.382789 4952 cpu_manager.go:414] "containerMap: removing container" podUID="3f222119-3b8c-4f8a-8ba2-88ef4d873da4" containerName="istio-proxy" Oct 30 23:40:53.382911 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:53.382797 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f222119-3b8c-4f8a-8ba2-88ef4d873da4" containerName="istio-proxy" Oct 30 23:40:53.383169 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:53.382957 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f222119-3b8c-4f8a-8ba2-88ef4d873da4" containerName="istio-proxy" Oct 30 23:40:53.383169 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:53.382975 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ac8fce7-3841-4fb4-a7a3-06359477953b" containerName="istio-proxy" Oct 30 23:40:53.383169 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:53.382993 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ad4e638-0d0d-423b-96db-f671e8095f08" containerName="istio-proxy" Oct 30 23:40:53.386901 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:53.386881 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-jc5dz" Oct 30 23:40:53.441657 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:53.441629 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-jc5dz"] Oct 30 23:40:53.455439 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:53.455404 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swmz7\" (UniqueName: \"kubernetes.io/projected/b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4-kube-api-access-swmz7\") pod \"payload-pre-processing-e2e-mt-55666687-84b5d7b947-jc5dz\" (UID: \"b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-jc5dz" Oct 30 23:40:53.455577 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:53.455543 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-55666687-84b5d7b947-jc5dz\" (UID: \"b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-jc5dz" Oct 30 23:40:53.486293 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:53.486269 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-55666687\"" Oct 30 23:40:53.486446 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:53.486428 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-55666687-dockercfg-ccbfh\"" Oct 30 23:40:53.556980 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:53.556948 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swmz7\" (UniqueName: \"kubernetes.io/projected/b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4-kube-api-access-swmz7\") pod \"payload-pre-processing-e2e-mt-55666687-84b5d7b947-jc5dz\" (UID: \"b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-jc5dz" Oct 30 23:40:53.557131 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:53.557032 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-55666687-84b5d7b947-jc5dz\" (UID: \"b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-jc5dz" Oct 30 23:40:53.557743 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:53.557724 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-55666687-84b5d7b947-jc5dz\" (UID: \"b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-jc5dz" Oct 30 23:40:53.614016 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:53.613993 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swmz7\" (UniqueName: \"kubernetes.io/projected/b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4-kube-api-access-swmz7\") pod \"payload-pre-processing-e2e-mt-55666687-84b5d7b947-jc5dz\" (UID: \"b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-jc5dz" Oct 30 23:40:53.703887 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:53.703828 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-jc5dz" Oct 30 23:40:54.082940 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:54.082910 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-jc5dz"] Oct 30 23:40:54.083538 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:40:54.083516 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8636dbc_385a_4b9f_a8ff_f0c40a87a6a4.slice/crio-e0e534f3ae0cf97d5cc723dc5e8e2c98be0a8a1479a7477828a81f6ab9a32a85 WatchSource:0}: Error finding container e0e534f3ae0cf97d5cc723dc5e8e2c98be0a8a1479a7477828a81f6ab9a32a85: Status 404 returned error can't find the container with id e0e534f3ae0cf97d5cc723dc5e8e2c98be0a8a1479a7477828a81f6ab9a32a85 Oct 30 23:40:54.289334 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:54.289296 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:54.340431 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:54.340394 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-jc5dz" event={"ID":"b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4","Type":"ContainerStarted","Data":"7a260691fcc10234db148b5f3584de3bc3e7491b516aa912dcb47c7b850959de"} Oct 30 23:40:54.340650 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:54.340464 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-jc5dz" event={"ID":"b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4","Type":"ContainerStarted","Data":"e0e534f3ae0cf97d5cc723dc5e8e2c98be0a8a1479a7477828a81f6ab9a32a85"} Oct 30 23:40:54.340650 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:54.340494 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-jc5dz" Oct 30 23:40:54.456621 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:54.456561 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-jc5dz" podStartSLOduration=1.456542529 podStartE2EDuration="1.456542529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:54.453481159 +0000 UTC m=+2073.597311388" watchObservedRunningTime="2026-10-30 23:40:54.456542529 +0000 UTC m=+2073.600372735" Oct 30 23:40:54.514097 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:54.514068 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:40:56.399465 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:40:56.399415 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:01.299405 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:01.299367 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:05.101573 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:05.101536 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:05.328876 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:05.328840 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:05.364365 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:05.364287 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-jc5dz" Oct 30 23:41:06.571131 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:06.571090 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:11.519890 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:11.519860 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6"] Oct 30 23:41:11.524338 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:11.524320 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6" Oct 30 23:41:11.616098 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:11.616025 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e27e01-b433-41c6-b19c-657bcb771e1c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6\" (UID: \"88e27e01-b433-41c6-b19c-657bcb771e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6" Oct 30 23:41:11.616297 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:11.616168 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88e27e01-b433-41c6-b19c-657bcb771e1c-maas-api-tls\") pod \"maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6\" (UID: \"88e27e01-b433-41c6-b19c-657bcb771e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6" Oct 30 23:41:11.616297 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:11.616204 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9v6t\" (UniqueName: \"kubernetes.io/projected/88e27e01-b433-41c6-b19c-657bcb771e1c-kube-api-access-j9v6t\") pod \"maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6\" (UID: \"88e27e01-b433-41c6-b19c-657bcb771e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6" Oct 30 23:41:11.616509 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:11.616487 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88e27e01-b433-41c6-b19c-657bcb771e1c-metrics-certs\") pod \"maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6\" (UID: \"88e27e01-b433-41c6-b19c-657bcb771e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6" Oct 30 23:41:11.626648 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:11.626626 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-7d1536-serving-cert\"" Oct 30 23:41:11.630582 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:11.630562 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6"] Oct 30 23:41:11.718409 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:11.718372 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88e27e01-b433-41c6-b19c-657bcb771e1c-metrics-certs\") pod \"maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6\" (UID: \"88e27e01-b433-41c6-b19c-657bcb771e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6" Oct 30 23:41:11.718601 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:11.718438 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e27e01-b433-41c6-b19c-657bcb771e1c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6\" (UID: \"88e27e01-b433-41c6-b19c-657bcb771e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6" Oct 30 23:41:11.718601 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:11.718486 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88e27e01-b433-41c6-b19c-657bcb771e1c-maas-api-tls\") pod \"maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6\" (UID: \"88e27e01-b433-41c6-b19c-657bcb771e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6" Oct 30 23:41:11.718601 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:11.718511 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9v6t\" (UniqueName: \"kubernetes.io/projected/88e27e01-b433-41c6-b19c-657bcb771e1c-kube-api-access-j9v6t\") pod \"maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6\" (UID: \"88e27e01-b433-41c6-b19c-657bcb771e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6" Oct 30 23:41:11.718785 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:41:11.718600 4952 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-7d1536-serving-cert: secret "maas-api-e2e-inf-a-7d1536-serving-cert" not found Oct 30 23:41:11.718785 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:41:11.718680 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88e27e01-b433-41c6-b19c-657bcb771e1c-maas-api-tls podName:88e27e01-b433-41c6-b19c-657bcb771e1c nodeName:}" failed. No retries permitted until 2026-10-30 23:41:12.218662792 +0000 UTC m=+2091.362492975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/88e27e01-b433-41c6-b19c-657bcb771e1c-maas-api-tls") pod "maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6" (UID: "88e27e01-b433-41c6-b19c-657bcb771e1c") : secret "maas-api-e2e-inf-a-7d1536-serving-cert" not found Oct 30 23:41:11.719298 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:11.719274 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e27e01-b433-41c6-b19c-657bcb771e1c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6\" (UID: \"88e27e01-b433-41c6-b19c-657bcb771e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6" Oct 30 23:41:11.721076 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:11.721023 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88e27e01-b433-41c6-b19c-657bcb771e1c-metrics-certs\") pod \"maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6\" (UID: \"88e27e01-b433-41c6-b19c-657bcb771e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6" Oct 30 23:41:11.783801 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:11.783678 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9v6t\" (UniqueName: \"kubernetes.io/projected/88e27e01-b433-41c6-b19c-657bcb771e1c-kube-api-access-j9v6t\") pod \"maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6\" (UID: \"88e27e01-b433-41c6-b19c-657bcb771e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6" Oct 30 23:41:11.869002 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:11.868935 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:12.104293 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:12.104232 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:12.225418 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:12.225383 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88e27e01-b433-41c6-b19c-657bcb771e1c-maas-api-tls\") pod \"maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6\" (UID: \"88e27e01-b433-41c6-b19c-657bcb771e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6" Oct 30 23:41:12.228226 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:12.228196 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88e27e01-b433-41c6-b19c-657bcb771e1c-maas-api-tls\") pod \"maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6\" (UID: \"88e27e01-b433-41c6-b19c-657bcb771e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6" Oct 30 23:41:12.355459 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:12.355372 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:12.439178 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:12.439149 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6" Oct 30 23:41:12.823734 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:41:12.823704 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e27e01_b433_41c6_b19c_657bcb771e1c.slice/crio-ad8e675b6375bfaddae9e684dd304428845bea544339e0e1ab459030aaa82f56 WatchSource:0}: Error finding container ad8e675b6375bfaddae9e684dd304428845bea544339e0e1ab459030aaa82f56: Status 404 returned error can't find the container with id ad8e675b6375bfaddae9e684dd304428845bea544339e0e1ab459030aaa82f56 Oct 30 23:41:12.824172 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:12.823946 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6"] Oct 30 23:41:13.552497 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:13.552463 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6" event={"ID":"88e27e01-b433-41c6-b19c-657bcb771e1c","Type":"ContainerStarted","Data":"ad8e675b6375bfaddae9e684dd304428845bea544339e0e1ab459030aaa82f56"} Oct 30 23:41:14.044264 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:14.037779 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:14.272507 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:14.272472 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:14.514362 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:14.514333 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:14.562976 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:14.562939 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6" event={"ID":"88e27e01-b433-41c6-b19c-657bcb771e1c","Type":"ContainerStarted","Data":"7d62cde5dd23b8094334e3708fdf4c3589182be864b439c9c4437f5981399029"} Oct 30 23:41:14.563167 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:14.563005 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6" Oct 30 23:41:14.686630 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:14.686576 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6" podStartSLOduration=2.868595464 podStartE2EDuration="3.686560841s" totalImagesPullingTime="817.965377ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:41:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:14.684698148 +0000 UTC m=+2093.828528355" watchObservedRunningTime="2026-10-30 23:41:14.686560841 +0000 UTC m=+2093.830391045" Oct 30 23:41:15.737096 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:15.737067 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:17.434229 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:17.434185 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:20.461134 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:20.461075 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:20.576382 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:20.576353 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6" Oct 30 23:41:21.143083 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:41:21.143029 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:41:21.306050 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:21.306005 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft_91651ce2-4673-4ced-b26d-e94ff8829d11/extract/0.log" Oct 30 23:41:21.315001 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:21.314965 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft_91651ce2-4673-4ced-b26d-e94ff8829d11/extract/0.log" Oct 30 23:41:21.317171 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:21.317152 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/2.log" Oct 30 23:41:21.323169 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:21.323150 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 30 23:41:21.325945 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:21.325928 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/2.log" Oct 30 23:41:21.331840 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:21.331821 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 30 23:41:21.920082 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:21.920031 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v"] Oct 30 23:41:21.925607 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:21.925590 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v" Oct 30 23:41:22.027387 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:22.027352 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-0a7530-serving-cert\"" Oct 30 23:41:22.031420 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:22.031398 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v"] Oct 30 23:41:22.065942 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:22.065911 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04-metrics-certs\") pod \"maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v\" (UID: \"bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v" Oct 30 23:41:22.066121 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:22.066063 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04-openshift-service-ca\") pod \"maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v\" (UID: \"bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v" Oct 30 23:41:22.066263 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:22.066241 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04-maas-api-tls\") pod \"maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v\" (UID: \"bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v" Oct 30 23:41:22.066328 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:22.066311 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8rkb\" (UniqueName: \"kubernetes.io/projected/bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04-kube-api-access-z8rkb\") pod \"maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v\" (UID: \"bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v" Oct 30 23:41:22.176150 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:22.168393 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04-metrics-certs\") pod \"maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v\" (UID: \"bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v" Oct 30 23:41:22.176150 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:22.168474 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04-openshift-service-ca\") pod \"maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v\" (UID: \"bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v" Oct 30 23:41:22.176150 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:22.168618 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04-maas-api-tls\") pod \"maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v\" (UID: \"bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v" Oct 30 23:41:22.176150 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:22.168683 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8rkb\" (UniqueName: \"kubernetes.io/projected/bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04-kube-api-access-z8rkb\") pod \"maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v\" (UID: \"bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v" Oct 30 23:41:22.176150 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:41:22.171106 4952 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-0a7530-serving-cert: secret "maas-api-e2e-api-a-0a7530-serving-cert" not found Oct 30 23:41:22.176150 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:41:22.171190 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04-maas-api-tls podName:bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:22.671167566 +0000 UTC m=+2101.814997759 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04-maas-api-tls") pod "maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v" (UID: "bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04") : secret "maas-api-e2e-api-a-0a7530-serving-cert" not found Oct 30 23:41:22.176150 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:22.172439 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04-openshift-service-ca\") pod \"maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v\" (UID: \"bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v" Oct 30 23:41:22.176150 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:22.172929 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04-metrics-certs\") pod \"maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v\" (UID: \"bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v" Oct 30 23:41:22.202933 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:22.202894 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:22.229749 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:22.229715 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8rkb\" (UniqueName: \"kubernetes.io/projected/bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04-kube-api-access-z8rkb\") pod \"maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v\" (UID: \"bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v" Oct 30 23:41:22.424247 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:22.424206 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:22.638909 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:22.638875 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:22.676313 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:22.676280 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04-maas-api-tls\") pod \"maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v\" (UID: \"bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v" Oct 30 23:41:22.678641 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:22.678617 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04-maas-api-tls\") pod \"maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v\" (UID: \"bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v" Oct 30 23:41:22.840160 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:22.840135 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v" Oct 30 23:41:23.221517 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:41:23.221487 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb5d297e_1e95_4e95_b2d5_7c8aad4e7a04.slice/crio-aef6cbcc30b05fa90590c6cc3431da7bfeb16bf882d5ac1e1478a0fa4f0e4878 WatchSource:0}: Error finding container aef6cbcc30b05fa90590c6cc3431da7bfeb16bf882d5ac1e1478a0fa4f0e4878: Status 404 returned error can't find the container with id aef6cbcc30b05fa90590c6cc3431da7bfeb16bf882d5ac1e1478a0fa4f0e4878 Oct 30 23:41:23.222528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:23.222499 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v"] Oct 30 23:41:23.631306 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:23.631272 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v" event={"ID":"bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04","Type":"ContainerStarted","Data":"aef6cbcc30b05fa90590c6cc3431da7bfeb16bf882d5ac1e1478a0fa4f0e4878"} Oct 30 23:41:24.639576 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:24.639540 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v" event={"ID":"bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04","Type":"ContainerStarted","Data":"baab6798f685b58b9629c815361cc82e4b4aca4b8b30f298d467ba535496b3f7"} Oct 30 23:41:24.640067 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:24.639635 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v" Oct 30 23:41:24.754633 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:24.754574 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v" podStartSLOduration=2.996756694 podStartE2EDuration="3.754557853s" totalImagesPullingTime="757.801159ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:24.754277309 +0000 UTC m=+2103.898107517" watchObservedRunningTime="2026-10-30 23:41:24.754557853 +0000 UTC m=+2103.898388058" Oct 30 23:41:27.184300 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:27.184260 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:30.654296 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:30.654215 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v" Oct 30 23:41:31.968290 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:31.968249 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:35.616128 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:35.616086 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:41:36.719092 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:36.719031 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:41.455233 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:41.455183 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:46.303917 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:46.303884 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:50.728467 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:50.728432 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8"] Oct 30 23:41:50.732696 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:50.732678 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8" Oct 30 23:41:50.785465 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:50.785425 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-26ce6b-serving-cert\"" Oct 30 23:41:50.789721 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:50.789697 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8"] Oct 30 23:41:50.854445 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:50.854409 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09eac6de-60af-424f-92e6-b230e63a3712-openshift-service-ca\") pod \"maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8\" (UID: \"09eac6de-60af-424f-92e6-b230e63a3712\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8" Oct 30 23:41:50.854771 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:50.854735 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09eac6de-60af-424f-92e6-b230e63a3712-metrics-certs\") pod \"maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8\" (UID: \"09eac6de-60af-424f-92e6-b230e63a3712\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8" Oct 30 23:41:50.854918 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:50.854900 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09eac6de-60af-424f-92e6-b230e63a3712-maas-api-tls\") pod \"maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8\" (UID: \"09eac6de-60af-424f-92e6-b230e63a3712\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8" Oct 30 23:41:50.855083 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:50.855065 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5x94\" (UniqueName: \"kubernetes.io/projected/09eac6de-60af-424f-92e6-b230e63a3712-kube-api-access-k5x94\") pod \"maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8\" (UID: \"09eac6de-60af-424f-92e6-b230e63a3712\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8" Oct 30 23:41:50.956264 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:50.956228 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5x94\" (UniqueName: \"kubernetes.io/projected/09eac6de-60af-424f-92e6-b230e63a3712-kube-api-access-k5x94\") pod \"maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8\" (UID: \"09eac6de-60af-424f-92e6-b230e63a3712\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8" Oct 30 23:41:50.956410 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:50.956292 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09eac6de-60af-424f-92e6-b230e63a3712-openshift-service-ca\") pod \"maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8\" (UID: \"09eac6de-60af-424f-92e6-b230e63a3712\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8" Oct 30 23:41:50.956410 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:50.956326 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09eac6de-60af-424f-92e6-b230e63a3712-metrics-certs\") pod \"maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8\" (UID: \"09eac6de-60af-424f-92e6-b230e63a3712\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8" Oct 30 23:41:50.956497 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:50.956423 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09eac6de-60af-424f-92e6-b230e63a3712-maas-api-tls\") pod \"maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8\" (UID: \"09eac6de-60af-424f-92e6-b230e63a3712\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8" Oct 30 23:41:50.956548 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:41:50.956534 4952 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-26ce6b-serving-cert: secret "maas-api-e2e-api-b-26ce6b-serving-cert" not found Oct 30 23:41:50.956593 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:41:50.956584 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09eac6de-60af-424f-92e6-b230e63a3712-maas-api-tls podName:09eac6de-60af-424f-92e6-b230e63a3712 nodeName:}" failed. No retries permitted until 2026-10-30 23:41:51.4565703 +0000 UTC m=+2130.600400496 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/09eac6de-60af-424f-92e6-b230e63a3712-maas-api-tls") pod "maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8" (UID: "09eac6de-60af-424f-92e6-b230e63a3712") : secret "maas-api-e2e-api-b-26ce6b-serving-cert" not found Oct 30 23:41:50.957103 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:50.957081 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09eac6de-60af-424f-92e6-b230e63a3712-openshift-service-ca\") pod \"maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8\" (UID: \"09eac6de-60af-424f-92e6-b230e63a3712\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8" Oct 30 23:41:50.958963 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:50.958942 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09eac6de-60af-424f-92e6-b230e63a3712-metrics-certs\") pod \"maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8\" (UID: \"09eac6de-60af-424f-92e6-b230e63a3712\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8" Oct 30 23:41:51.016016 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:51.015948 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5x94\" (UniqueName: \"kubernetes.io/projected/09eac6de-60af-424f-92e6-b230e63a3712-kube-api-access-k5x94\") pod \"maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8\" (UID: \"09eac6de-60af-424f-92e6-b230e63a3712\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8" Oct 30 23:41:51.078345 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:51.078320 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:51.166091 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:41:51.165580 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:41:51.379845 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:51.379816 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:51.466339 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:51.466305 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09eac6de-60af-424f-92e6-b230e63a3712-maas-api-tls\") pod \"maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8\" (UID: \"09eac6de-60af-424f-92e6-b230e63a3712\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8" Oct 30 23:41:51.469023 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:51.468991 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09eac6de-60af-424f-92e6-b230e63a3712-maas-api-tls\") pod \"maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8\" (UID: \"09eac6de-60af-424f-92e6-b230e63a3712\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8" Oct 30 23:41:51.647341 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:51.647266 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8" Oct 30 23:41:52.026760 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:52.026733 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8"] Oct 30 23:41:52.027846 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:41:52.027820 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09eac6de_60af_424f_92e6_b230e63a3712.slice/crio-0a205cb7a72f8ba40abba290673bf2c7052a76f94964cd81c3641aaf0c823c0b WatchSource:0}: Error finding container 0a205cb7a72f8ba40abba290673bf2c7052a76f94964cd81c3641aaf0c823c0b: Status 404 returned error can't find the container with id 0a205cb7a72f8ba40abba290673bf2c7052a76f94964cd81c3641aaf0c823c0b Oct 30 23:41:52.865367 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:52.865319 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8" event={"ID":"09eac6de-60af-424f-92e6-b230e63a3712","Type":"ContainerStarted","Data":"0a205cb7a72f8ba40abba290673bf2c7052a76f94964cd81c3641aaf0c823c0b"} Oct 30 23:41:53.874561 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:53.874518 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8" event={"ID":"09eac6de-60af-424f-92e6-b230e63a3712","Type":"ContainerStarted","Data":"c8529780cf2210292279647210b8e7c0d0271e2f81f5480a36cf398d6b6d3056"} Oct 30 23:41:53.875082 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:53.874594 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8" Oct 30 23:41:53.989321 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:53.989124 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8" podStartSLOduration=3.146636656 podStartE2EDuration="3.989098126s" totalImagesPullingTime="842.46147ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:41:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:53.988867622 +0000 UTC m=+2133.132697829" watchObservedRunningTime="2026-10-30 23:41:53.989098126 +0000 UTC m=+2133.132928335" Oct 30 23:41:55.969380 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:55.969351 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:41:59.887301 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:41:59.887274 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8" Oct 30 23:42:00.835344 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:00.835303 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:05.591768 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:05.591720 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:09.609550 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:09.609514 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8"] Oct 30 23:42:09.610029 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:09.609778 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8" podUID="164790dc-301c-40a2-a62a-665146b9adc2" containerName="maas-api" containerID="cri-o://8f4ce9805e58c1343efc67f5ee2441faffa72471c612af1e3c3901036025f4c0" gracePeriod=30 Oct 30 23:42:09.905843 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:09.905745 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:09.956087 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:09.956057 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8" Oct 30 23:42:10.022090 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:10.008235 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/164790dc-301c-40a2-a62a-665146b9adc2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/164790dc-301c-40a2-a62a-665146b9adc2-openshift-service-ca\") pod \"164790dc-301c-40a2-a62a-665146b9adc2\" (UID: \"164790dc-301c-40a2-a62a-665146b9adc2\") " Oct 30 23:42:10.022090 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:10.008521 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/164790dc-301c-40a2-a62a-665146b9adc2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/164790dc-301c-40a2-a62a-665146b9adc2-metrics-certs\") pod \"164790dc-301c-40a2-a62a-665146b9adc2\" (UID: \"164790dc-301c-40a2-a62a-665146b9adc2\") " Oct 30 23:42:10.022090 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:10.008581 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/164790dc-301c-40a2-a62a-665146b9adc2-kube-api-access-bhkh5\" (UniqueName: \"kubernetes.io/projected/164790dc-301c-40a2-a62a-665146b9adc2-kube-api-access-bhkh5\") pod \"164790dc-301c-40a2-a62a-665146b9adc2\" (UID: \"164790dc-301c-40a2-a62a-665146b9adc2\") " Oct 30 23:42:10.022090 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:10.008654 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/164790dc-301c-40a2-a62a-665146b9adc2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/164790dc-301c-40a2-a62a-665146b9adc2-maas-api-tls\") pod \"164790dc-301c-40a2-a62a-665146b9adc2\" (UID: \"164790dc-301c-40a2-a62a-665146b9adc2\") " Oct 30 23:42:10.022090 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:10.016248 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/164790dc-301c-40a2-a62a-665146b9adc2-openshift-service-ca" pod "164790dc-301c-40a2-a62a-665146b9adc2" (UID: "164790dc-301c-40a2-a62a-665146b9adc2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:42:10.031170 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:10.029389 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/164790dc-301c-40a2-a62a-665146b9adc2-metrics-certs" pod "164790dc-301c-40a2-a62a-665146b9adc2" (UID: "164790dc-301c-40a2-a62a-665146b9adc2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:42:10.035319 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:10.035285 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/164790dc-301c-40a2-a62a-665146b9adc2-maas-api-tls" pod "164790dc-301c-40a2-a62a-665146b9adc2" (UID: "164790dc-301c-40a2-a62a-665146b9adc2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:42:10.047020 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:10.046471 4952 generic.go:356] "Generic (PLEG): container finished" podID="164790dc-301c-40a2-a62a-665146b9adc2" containerID="8f4ce9805e58c1343efc67f5ee2441faffa72471c612af1e3c3901036025f4c0" exitCode=0 Oct 30 23:42:10.047020 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:10.046613 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8" event={"ID":"164790dc-301c-40a2-a62a-665146b9adc2","Type":"ContainerDied","Data":"8f4ce9805e58c1343efc67f5ee2441faffa72471c612af1e3c3901036025f4c0"} Oct 30 23:42:10.047020 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:10.046664 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8" event={"ID":"164790dc-301c-40a2-a62a-665146b9adc2","Type":"ContainerDied","Data":"9c22f3778252c03f23598a2975c9d14d4e1cf0ff21a96a89c545beb07a2adb57"} Oct 30 23:42:10.047020 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:10.046694 4952 scope.go:122] "RemoveContainer" containerID="8f4ce9805e58c1343efc67f5ee2441faffa72471c612af1e3c3901036025f4c0" Oct 30 23:42:10.047020 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:10.046880 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8" Oct 30 23:42:10.048789 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:10.048746 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/164790dc-301c-40a2-a62a-665146b9adc2-kube-api-access-bhkh5" pod "164790dc-301c-40a2-a62a-665146b9adc2" (UID: "164790dc-301c-40a2-a62a-665146b9adc2"). InnerVolumeSpecName "kube-api-access-bhkh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:42:10.069678 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:10.069649 4952 scope.go:122] "RemoveContainer" containerID="8f4ce9805e58c1343efc67f5ee2441faffa72471c612af1e3c3901036025f4c0" Oct 30 23:42:10.070229 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:42:10.070201 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f4ce9805e58c1343efc67f5ee2441faffa72471c612af1e3c3901036025f4c0\": container with ID starting with 8f4ce9805e58c1343efc67f5ee2441faffa72471c612af1e3c3901036025f4c0 not found: ID does not exist" containerID="8f4ce9805e58c1343efc67f5ee2441faffa72471c612af1e3c3901036025f4c0" Oct 30 23:42:10.070334 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:10.070244 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f4ce9805e58c1343efc67f5ee2441faffa72471c612af1e3c3901036025f4c0"} err="failed to get container status \"8f4ce9805e58c1343efc67f5ee2441faffa72471c612af1e3c3901036025f4c0\": rpc error: code = NotFound desc = could not find container \"8f4ce9805e58c1343efc67f5ee2441faffa72471c612af1e3c3901036025f4c0\": container with ID starting with 8f4ce9805e58c1343efc67f5ee2441faffa72471c612af1e3c3901036025f4c0 not found: ID does not exist" Oct 30 23:42:10.115140 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:10.115094 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/164790dc-301c-40a2-a62a-665146b9adc2-metrics-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:10.115140 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:10.115133 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhkh5\" (UniqueName: \"kubernetes.io/projected/164790dc-301c-40a2-a62a-665146b9adc2-kube-api-access-bhkh5\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:10.115421 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:10.115149 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/164790dc-301c-40a2-a62a-665146b9adc2-maas-api-tls\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:10.115421 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:10.115163 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/164790dc-301c-40a2-a62a-665146b9adc2-openshift-service-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:10.469941 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:10.469911 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8"] Oct 30 23:42:10.473514 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:10.473489 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b526ac-6b58855585-lbqr8"] Oct 30 23:42:10.972130 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:10.972099 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="164790dc-301c-40a2-a62a-665146b9adc2" path="/var/lib/kubelet/pods/164790dc-301c-40a2-a62a-665146b9adc2/volumes" Oct 30 23:42:11.361754 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:11.361720 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:11.605360 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:11.605321 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:11.859019 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:11.858960 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:12.271262 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:12.271148 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:16.327561 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:16.327528 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-55666687-7659595c45-jwcgn"] Oct 30 23:42:16.328175 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:16.328161 4952 cpu_manager.go:414] "containerMap: removing container" podUID="164790dc-301c-40a2-a62a-665146b9adc2" containerName="maas-api" Oct 30 23:42:16.328230 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:16.328176 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="164790dc-301c-40a2-a62a-665146b9adc2" containerName="maas-api" Oct 30 23:42:16.328280 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:16.328270 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="164790dc-301c-40a2-a62a-665146b9adc2" containerName="maas-api" Oct 30 23:42:16.331899 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:16.331879 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55666687-7659595c45-jwcgn" Oct 30 23:42:16.388881 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:16.388850 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f0f494d-1bc3-4981-8203-4adb0d1b0340-metrics-certs\") pod \"maas-api-e2e-mt-55666687-7659595c45-jwcgn\" (UID: \"8f0f494d-1bc3-4981-8203-4adb0d1b0340\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55666687-7659595c45-jwcgn" Oct 30 23:42:16.389015 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:16.388897 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f0f494d-1bc3-4981-8203-4adb0d1b0340-openshift-service-ca\") pod \"maas-api-e2e-mt-55666687-7659595c45-jwcgn\" (UID: \"8f0f494d-1bc3-4981-8203-4adb0d1b0340\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55666687-7659595c45-jwcgn" Oct 30 23:42:16.389099 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:16.389078 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzfsq\" (UniqueName: \"kubernetes.io/projected/8f0f494d-1bc3-4981-8203-4adb0d1b0340-kube-api-access-fzfsq\") pod \"maas-api-e2e-mt-55666687-7659595c45-jwcgn\" (UID: \"8f0f494d-1bc3-4981-8203-4adb0d1b0340\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55666687-7659595c45-jwcgn" Oct 30 23:42:16.389192 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:16.389168 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f0f494d-1bc3-4981-8203-4adb0d1b0340-maas-api-tls\") pod \"maas-api-e2e-mt-55666687-7659595c45-jwcgn\" (UID: \"8f0f494d-1bc3-4981-8203-4adb0d1b0340\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55666687-7659595c45-jwcgn" Oct 30 23:42:16.435508 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:16.435482 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-55666687-serving-cert\"" Oct 30 23:42:16.439596 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:16.439574 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-55666687-7659595c45-jwcgn"] Oct 30 23:42:16.490534 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:16.490502 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f0f494d-1bc3-4981-8203-4adb0d1b0340-metrics-certs\") pod \"maas-api-e2e-mt-55666687-7659595c45-jwcgn\" (UID: \"8f0f494d-1bc3-4981-8203-4adb0d1b0340\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55666687-7659595c45-jwcgn" Oct 30 23:42:16.490690 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:16.490552 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f0f494d-1bc3-4981-8203-4adb0d1b0340-openshift-service-ca\") pod \"maas-api-e2e-mt-55666687-7659595c45-jwcgn\" (UID: \"8f0f494d-1bc3-4981-8203-4adb0d1b0340\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55666687-7659595c45-jwcgn" Oct 30 23:42:16.490690 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:16.490669 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzfsq\" (UniqueName: \"kubernetes.io/projected/8f0f494d-1bc3-4981-8203-4adb0d1b0340-kube-api-access-fzfsq\") pod \"maas-api-e2e-mt-55666687-7659595c45-jwcgn\" (UID: \"8f0f494d-1bc3-4981-8203-4adb0d1b0340\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55666687-7659595c45-jwcgn" Oct 30 23:42:16.490823 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:16.490749 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f0f494d-1bc3-4981-8203-4adb0d1b0340-maas-api-tls\") pod \"maas-api-e2e-mt-55666687-7659595c45-jwcgn\" (UID: \"8f0f494d-1bc3-4981-8203-4adb0d1b0340\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55666687-7659595c45-jwcgn" Oct 30 23:42:16.491013 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:42:16.490987 4952 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-55666687-serving-cert: secret "maas-api-e2e-mt-55666687-serving-cert" not found Oct 30 23:42:16.491164 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:42:16.491094 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f0f494d-1bc3-4981-8203-4adb0d1b0340-maas-api-tls podName:8f0f494d-1bc3-4981-8203-4adb0d1b0340 nodeName:}" failed. No retries permitted until 2026-10-30 23:42:16.991073561 +0000 UTC m=+2156.134903750 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8f0f494d-1bc3-4981-8203-4adb0d1b0340-maas-api-tls") pod "maas-api-e2e-mt-55666687-7659595c45-jwcgn" (UID: "8f0f494d-1bc3-4981-8203-4adb0d1b0340") : secret "maas-api-e2e-mt-55666687-serving-cert" not found Oct 30 23:42:16.492687 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:16.492664 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f0f494d-1bc3-4981-8203-4adb0d1b0340-openshift-service-ca\") pod \"maas-api-e2e-mt-55666687-7659595c45-jwcgn\" (UID: \"8f0f494d-1bc3-4981-8203-4adb0d1b0340\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55666687-7659595c45-jwcgn" Oct 30 23:42:16.494248 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:16.494224 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f0f494d-1bc3-4981-8203-4adb0d1b0340-metrics-certs\") pod \"maas-api-e2e-mt-55666687-7659595c45-jwcgn\" (UID: \"8f0f494d-1bc3-4981-8203-4adb0d1b0340\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55666687-7659595c45-jwcgn" Oct 30 23:42:16.549272 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:16.549248 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzfsq\" (UniqueName: \"kubernetes.io/projected/8f0f494d-1bc3-4981-8203-4adb0d1b0340-kube-api-access-fzfsq\") pod \"maas-api-e2e-mt-55666687-7659595c45-jwcgn\" (UID: \"8f0f494d-1bc3-4981-8203-4adb0d1b0340\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55666687-7659595c45-jwcgn" Oct 30 23:42:16.621579 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:16.621520 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:16.850248 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:16.850217 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:17.000472 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:17.000398 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f0f494d-1bc3-4981-8203-4adb0d1b0340-maas-api-tls\") pod \"maas-api-e2e-mt-55666687-7659595c45-jwcgn\" (UID: \"8f0f494d-1bc3-4981-8203-4adb0d1b0340\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55666687-7659595c45-jwcgn" Oct 30 23:42:17.002982 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:17.002961 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f0f494d-1bc3-4981-8203-4adb0d1b0340-maas-api-tls\") pod \"maas-api-e2e-mt-55666687-7659595c45-jwcgn\" (UID: \"8f0f494d-1bc3-4981-8203-4adb0d1b0340\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55666687-7659595c45-jwcgn" Oct 30 23:42:17.093245 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:17.093191 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:17.246694 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:17.246660 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55666687-7659595c45-jwcgn" Oct 30 23:42:17.630934 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:17.630908 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-55666687-7659595c45-jwcgn"] Oct 30 23:42:17.631281 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:42:17.630989 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f0f494d_1bc3_4981_8203_4adb0d1b0340.slice/crio-a2116525a88a3d9d7f5c00971c66aafbcbd9b92561df90a7e4a14e666ec83a11 WatchSource:0}: Error finding container a2116525a88a3d9d7f5c00971c66aafbcbd9b92561df90a7e4a14e666ec83a11: Status 404 returned error can't find the container with id a2116525a88a3d9d7f5c00971c66aafbcbd9b92561df90a7e4a14e666ec83a11 Oct 30 23:42:18.107101 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:18.107068 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55666687-7659595c45-jwcgn" event={"ID":"8f0f494d-1bc3-4981-8203-4adb0d1b0340","Type":"ContainerStarted","Data":"a2116525a88a3d9d7f5c00971c66aafbcbd9b92561df90a7e4a14e666ec83a11"} Oct 30 23:42:19.115951 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:19.115916 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55666687-7659595c45-jwcgn" event={"ID":"8f0f494d-1bc3-4981-8203-4adb0d1b0340","Type":"ContainerStarted","Data":"cd44f411eba882c4bc1f492dd926c35d21629a4cdfebe402105dc7699b8d2c9e"} Oct 30 23:42:19.116333 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:19.116003 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55666687-7659595c45-jwcgn" Oct 30 23:42:19.206341 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:19.206308 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:19.232599 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:19.232545 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55666687-7659595c45-jwcgn" podStartSLOduration=2.404509141 podStartE2EDuration="3.232531739s" totalImagesPullingTime="828.022598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:42:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:19.230683256 +0000 UTC m=+2158.374513461" watchObservedRunningTime="2026-10-30 23:42:19.232531739 +0000 UTC m=+2158.376361944" Oct 30 23:42:19.441779 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:19.441648 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:19.755780 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:19.755674 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:21.146532 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:42:21.146490 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:42:21.341856 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:21.341820 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:22.792120 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:22.792025 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:23.041207 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:23.041176 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:23.275764 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:23.275726 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:23.506269 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:23.506241 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:23.811876 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:23.811843 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:25.130289 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:25.130261 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55666687-7659595c45-jwcgn" Oct 30 23:42:28.406579 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:28.406539 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:28.910465 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:28.910432 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:29.130299 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:29.130260 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:30.107637 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:30.107603 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:30.343331 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:30.343295 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:30.694262 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:30.694225 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:31.388494 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:31.388460 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:42:33.415449 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:33.415418 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g"] Oct 30 23:42:33.415908 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:33.415663 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" podUID="53756ab3-37fe-4d52-b197-7066f9b257c1" containerName="istio-proxy" containerID="cri-o://5ad7201c57b5e2cf62b3e2ea872c54c88482f26b5ab5b9d18dd0d91e9cb98f63" gracePeriod=30 Oct 30 23:42:33.575241 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:33.575191 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:34.307332 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:34.307295 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:34.543913 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:34.543881 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:35.203163 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:35.203129 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w"] Oct 30 23:42:35.203397 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:35.203372 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" podUID="9eb85bfe-fe64-4db7-abc4-6a453d8c3060" containerName="istio-proxy" containerID="cri-o://6212e45b4f25274285b0b8bf55c23a7bb4ac32dd5f61ad03d9d361cbd54a61d8" gracePeriod=30 Oct 30 23:42:35.550567 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:35.550475 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld"] Oct 30 23:42:35.551025 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:35.550780 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" podUID="3a7312d5-8075-4543-a074-1f176cf36911" containerName="istio-proxy" containerID="cri-o://49caa245a97c277c2ac3e396ce634547f91a811b7ea94d2039bded1c963c884b" gracePeriod=30 Oct 30 23:42:36.220308 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:36.220275 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2"] Oct 30 23:42:36.220530 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:36.220466 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" podUID="98027a1c-59f6-4d79-a118-dae8ed53e50b" containerName="istio-proxy" containerID="cri-o://a178f50b0fbcf4b2a7b8bf4b41a1c0b9750b7b8a9dd2a0b322ea02f8816d5c1e" gracePeriod=30 Oct 30 23:42:36.978432 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:36.978397 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 23:42:36.978884 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:36.978451 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" podUID="98027a1c-59f6-4d79-a118-dae8ed53e50b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:42:37.508670 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:37.508638 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-9ml7g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 23:42:37.508857 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:37.508694 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" podUID="53756ab3-37fe-4d52-b197-7066f9b257c1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:42:38.056439 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:38.056405 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:38.693615 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:38.693544 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" Oct 30 23:42:38.842057 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:38.841010 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53756ab3-37fe-4d52-b197-7066f9b257c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53756ab3-37fe-4d52-b197-7066f9b257c1-credential-socket\") pod \"53756ab3-37fe-4d52-b197-7066f9b257c1\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " Oct 30 23:42:38.842057 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:38.841175 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53756ab3-37fe-4d52-b197-7066f9b257c1-istio-token\" (UniqueName: \"kubernetes.io/projected/53756ab3-37fe-4d52-b197-7066f9b257c1-istio-token\") pod \"53756ab3-37fe-4d52-b197-7066f9b257c1\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " Oct 30 23:42:38.842057 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:38.841224 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53756ab3-37fe-4d52-b197-7066f9b257c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53756ab3-37fe-4d52-b197-7066f9b257c1-istio-data\") pod \"53756ab3-37fe-4d52-b197-7066f9b257c1\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " Oct 30 23:42:38.842057 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:38.841288 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53756ab3-37fe-4d52-b197-7066f9b257c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53756ab3-37fe-4d52-b197-7066f9b257c1-workload-certs\") pod \"53756ab3-37fe-4d52-b197-7066f9b257c1\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " Oct 30 23:42:38.842057 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:38.841393 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53756ab3-37fe-4d52-b197-7066f9b257c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53756ab3-37fe-4d52-b197-7066f9b257c1-istiod-ca-cert\") pod \"53756ab3-37fe-4d52-b197-7066f9b257c1\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " Oct 30 23:42:38.842057 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:38.841505 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53756ab3-37fe-4d52-b197-7066f9b257c1-kube-api-access-jzcgb\" (UniqueName: \"kubernetes.io/projected/53756ab3-37fe-4d52-b197-7066f9b257c1-kube-api-access-jzcgb\") pod \"53756ab3-37fe-4d52-b197-7066f9b257c1\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " Oct 30 23:42:38.842057 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:38.841568 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53756ab3-37fe-4d52-b197-7066f9b257c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53756ab3-37fe-4d52-b197-7066f9b257c1-istio-envoy\") pod \"53756ab3-37fe-4d52-b197-7066f9b257c1\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " Oct 30 23:42:38.842057 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:38.841618 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53756ab3-37fe-4d52-b197-7066f9b257c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53756ab3-37fe-4d52-b197-7066f9b257c1-istio-podinfo\") pod \"53756ab3-37fe-4d52-b197-7066f9b257c1\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " Oct 30 23:42:38.842057 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:38.841684 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53756ab3-37fe-4d52-b197-7066f9b257c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53756ab3-37fe-4d52-b197-7066f9b257c1-workload-socket\") pod \"53756ab3-37fe-4d52-b197-7066f9b257c1\" (UID: \"53756ab3-37fe-4d52-b197-7066f9b257c1\") " Oct 30 23:42:38.851166 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:38.845167 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53756ab3-37fe-4d52-b197-7066f9b257c1-workload-certs" pod "53756ab3-37fe-4d52-b197-7066f9b257c1" (UID: "53756ab3-37fe-4d52-b197-7066f9b257c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:42:38.865890 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:38.864009 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53756ab3-37fe-4d52-b197-7066f9b257c1-istio-data" pod "53756ab3-37fe-4d52-b197-7066f9b257c1" (UID: "53756ab3-37fe-4d52-b197-7066f9b257c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:42:38.865890 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:38.865262 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53756ab3-37fe-4d52-b197-7066f9b257c1-istiod-ca-cert" pod "53756ab3-37fe-4d52-b197-7066f9b257c1" (UID: "53756ab3-37fe-4d52-b197-7066f9b257c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:42:38.868017 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:38.867366 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53756ab3-37fe-4d52-b197-7066f9b257c1-credential-socket" pod "53756ab3-37fe-4d52-b197-7066f9b257c1" (UID: "53756ab3-37fe-4d52-b197-7066f9b257c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:42:38.868017 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:38.867527 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53756ab3-37fe-4d52-b197-7066f9b257c1-workload-socket" pod "53756ab3-37fe-4d52-b197-7066f9b257c1" (UID: "53756ab3-37fe-4d52-b197-7066f9b257c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:42:38.927500 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:38.890540 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53756ab3-37fe-4d52-b197-7066f9b257c1-istio-podinfo" pod "53756ab3-37fe-4d52-b197-7066f9b257c1" (UID: "53756ab3-37fe-4d52-b197-7066f9b257c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:42:38.927500 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:38.904398 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53756ab3-37fe-4d52-b197-7066f9b257c1-kube-api-access-jzcgb" pod "53756ab3-37fe-4d52-b197-7066f9b257c1" (UID: "53756ab3-37fe-4d52-b197-7066f9b257c1"). InnerVolumeSpecName "kube-api-access-jzcgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:42:38.927500 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:38.905179 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53756ab3-37fe-4d52-b197-7066f9b257c1-istio-token" pod "53756ab3-37fe-4d52-b197-7066f9b257c1" (UID: "53756ab3-37fe-4d52-b197-7066f9b257c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:42:38.950617 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:38.950459 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53756ab3-37fe-4d52-b197-7066f9b257c1-istio-token\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:38.950617 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:38.950496 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53756ab3-37fe-4d52-b197-7066f9b257c1-istio-data\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:38.950617 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:38.950494 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53756ab3-37fe-4d52-b197-7066f9b257c1-istio-envoy" pod "53756ab3-37fe-4d52-b197-7066f9b257c1" (UID: "53756ab3-37fe-4d52-b197-7066f9b257c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:42:38.950617 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:38.950511 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53756ab3-37fe-4d52-b197-7066f9b257c1-workload-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:38.950617 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:38.950527 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53756ab3-37fe-4d52-b197-7066f9b257c1-istiod-ca-cert\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:38.950617 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:38.950541 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzcgb\" (UniqueName: \"kubernetes.io/projected/53756ab3-37fe-4d52-b197-7066f9b257c1-kube-api-access-jzcgb\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:38.950617 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:38.950555 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53756ab3-37fe-4d52-b197-7066f9b257c1-istio-podinfo\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:38.950617 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:38.950570 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53756ab3-37fe-4d52-b197-7066f9b257c1-workload-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:38.950617 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:38.950584 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53756ab3-37fe-4d52-b197-7066f9b257c1-credential-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:39.052108 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:39.052072 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53756ab3-37fe-4d52-b197-7066f9b257c1-istio-envoy\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:39.286346 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:39.286262 4952 generic.go:356] "Generic (PLEG): container finished" podID="53756ab3-37fe-4d52-b197-7066f9b257c1" containerID="5ad7201c57b5e2cf62b3e2ea872c54c88482f26b5ab5b9d18dd0d91e9cb98f63" exitCode=0 Oct 30 23:42:39.286346 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:39.286304 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" Oct 30 23:42:39.286817 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:39.286344 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" event={"ID":"53756ab3-37fe-4d52-b197-7066f9b257c1","Type":"ContainerDied","Data":"5ad7201c57b5e2cf62b3e2ea872c54c88482f26b5ab5b9d18dd0d91e9cb98f63"} Oct 30 23:42:39.286817 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:39.286399 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g" event={"ID":"53756ab3-37fe-4d52-b197-7066f9b257c1","Type":"ContainerDied","Data":"aa9b43ed53597f476fc0598b656ebcccc998354daa2ff4fc9fc85f916a22f6ba"} Oct 30 23:42:39.286817 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:39.286422 4952 scope.go:122] "RemoveContainer" containerID="5ad7201c57b5e2cf62b3e2ea872c54c88482f26b5ab5b9d18dd0d91e9cb98f63" Oct 30 23:42:39.297562 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:39.297545 4952 scope.go:122] "RemoveContainer" containerID="5ad7201c57b5e2cf62b3e2ea872c54c88482f26b5ab5b9d18dd0d91e9cb98f63" Oct 30 23:42:39.297830 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:42:39.297810 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ad7201c57b5e2cf62b3e2ea872c54c88482f26b5ab5b9d18dd0d91e9cb98f63\": container with ID starting with 5ad7201c57b5e2cf62b3e2ea872c54c88482f26b5ab5b9d18dd0d91e9cb98f63 not found: ID does not exist" containerID="5ad7201c57b5e2cf62b3e2ea872c54c88482f26b5ab5b9d18dd0d91e9cb98f63" Oct 30 23:42:39.297899 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:39.297843 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ad7201c57b5e2cf62b3e2ea872c54c88482f26b5ab5b9d18dd0d91e9cb98f63"} err="failed to get container status \"5ad7201c57b5e2cf62b3e2ea872c54c88482f26b5ab5b9d18dd0d91e9cb98f63\": rpc error: code = NotFound desc = could not find container \"5ad7201c57b5e2cf62b3e2ea872c54c88482f26b5ab5b9d18dd0d91e9cb98f63\": container with ID starting with 5ad7201c57b5e2cf62b3e2ea872c54c88482f26b5ab5b9d18dd0d91e9cb98f63 not found: ID does not exist" Oct 30 23:42:39.456370 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:39.456006 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g"] Oct 30 23:42:39.460406 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:39.460381 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-9ml7g"] Oct 30 23:42:39.595443 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:39.595401 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:39.824436 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:39.824401 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:40.045130 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.044968 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:40.487515 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.487485 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" Oct 30 23:42:40.556572 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.556521 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:40.575134 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.575071 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-workload-socket\") pod \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " Oct 30 23:42:40.575134 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.575124 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-workload-certs\") pod \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " Oct 30 23:42:40.575352 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.575156 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-istio-token\" (UniqueName: \"kubernetes.io/projected/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-istio-token\") pod \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " Oct 30 23:42:40.575352 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.575233 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-credential-socket\") pod \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " Oct 30 23:42:40.575352 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.575296 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-kube-api-access-cmxw7\" (UniqueName: \"kubernetes.io/projected/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-kube-api-access-cmxw7\") pod \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " Oct 30 23:42:40.575649 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.575350 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-workload-socket" pod "9eb85bfe-fe64-4db7-abc4-6a453d8c3060" (UID: "9eb85bfe-fe64-4db7-abc4-6a453d8c3060"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:42:40.575649 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.575472 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-workload-certs" pod "9eb85bfe-fe64-4db7-abc4-6a453d8c3060" (UID: "9eb85bfe-fe64-4db7-abc4-6a453d8c3060"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:42:40.575649 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.575619 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-credential-socket" pod "9eb85bfe-fe64-4db7-abc4-6a453d8c3060" (UID: "9eb85bfe-fe64-4db7-abc4-6a453d8c3060"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:42:40.575789 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.575346 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-istio-envoy\") pod \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " Oct 30 23:42:40.575789 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.575781 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-istio-podinfo\") pod \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " Oct 30 23:42:40.575989 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.575970 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-istio-data\") pod \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " Oct 30 23:42:40.576090 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.576013 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-istiod-ca-cert\") pod \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\" (UID: \"9eb85bfe-fe64-4db7-abc4-6a453d8c3060\") " Oct 30 23:42:40.576613 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.576583 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-istio-data" pod "9eb85bfe-fe64-4db7-abc4-6a453d8c3060" (UID: "9eb85bfe-fe64-4db7-abc4-6a453d8c3060"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:42:40.577077 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.576912 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-istiod-ca-cert" pod "9eb85bfe-fe64-4db7-abc4-6a453d8c3060" (UID: "9eb85bfe-fe64-4db7-abc4-6a453d8c3060"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:42:40.577172 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.577004 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-istio-data\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:40.577172 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.577119 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-workload-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:40.577172 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.577137 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-workload-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:40.577172 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.577152 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-credential-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:40.578585 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.578542 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-istio-token" pod "9eb85bfe-fe64-4db7-abc4-6a453d8c3060" (UID: "9eb85bfe-fe64-4db7-abc4-6a453d8c3060"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:42:40.578760 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.578721 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-istio-envoy" pod "9eb85bfe-fe64-4db7-abc4-6a453d8c3060" (UID: "9eb85bfe-fe64-4db7-abc4-6a453d8c3060"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:42:40.578760 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.578732 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-kube-api-access-cmxw7" pod "9eb85bfe-fe64-4db7-abc4-6a453d8c3060" (UID: "9eb85bfe-fe64-4db7-abc4-6a453d8c3060"). InnerVolumeSpecName "kube-api-access-cmxw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:42:40.579795 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.579773 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-istio-podinfo" pod "9eb85bfe-fe64-4db7-abc4-6a453d8c3060" (UID: "9eb85bfe-fe64-4db7-abc4-6a453d8c3060"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:42:40.685082 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.684992 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-istiod-ca-cert\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:40.685082 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.685048 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-istio-token\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:40.685082 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.685065 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmxw7\" (UniqueName: \"kubernetes.io/projected/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-kube-api-access-cmxw7\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:40.685082 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.685093 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-istio-envoy\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:40.685376 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.685117 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb85bfe-fe64-4db7-abc4-6a453d8c3060-istio-podinfo\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:40.854562 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.854539 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" Oct 30 23:42:40.971146 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.971117 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53756ab3-37fe-4d52-b197-7066f9b257c1" path="/var/lib/kubelet/pods/53756ab3-37fe-4d52-b197-7066f9b257c1/volumes" Oct 30 23:42:40.988219 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.988194 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a7312d5-8075-4543-a074-1f176cf36911-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a7312d5-8075-4543-a074-1f176cf36911-istio-podinfo\") pod \"3a7312d5-8075-4543-a074-1f176cf36911\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " Oct 30 23:42:40.988345 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.988318 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a7312d5-8075-4543-a074-1f176cf36911-kube-api-access-htxb8\" (UniqueName: \"kubernetes.io/projected/3a7312d5-8075-4543-a074-1f176cf36911-kube-api-access-htxb8\") pod \"3a7312d5-8075-4543-a074-1f176cf36911\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " Oct 30 23:42:40.988410 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.988376 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a7312d5-8075-4543-a074-1f176cf36911-istio-token\" (UniqueName: \"kubernetes.io/projected/3a7312d5-8075-4543-a074-1f176cf36911-istio-token\") pod \"3a7312d5-8075-4543-a074-1f176cf36911\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " Oct 30 23:42:40.988468 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.988436 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a7312d5-8075-4543-a074-1f176cf36911-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7312d5-8075-4543-a074-1f176cf36911-credential-socket\") pod \"3a7312d5-8075-4543-a074-1f176cf36911\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " Oct 30 23:42:40.988524 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.988475 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a7312d5-8075-4543-a074-1f176cf36911-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a7312d5-8075-4543-a074-1f176cf36911-istiod-ca-cert\") pod \"3a7312d5-8075-4543-a074-1f176cf36911\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " Oct 30 23:42:40.988524 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.988507 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a7312d5-8075-4543-a074-1f176cf36911-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a7312d5-8075-4543-a074-1f176cf36911-istio-envoy\") pod \"3a7312d5-8075-4543-a074-1f176cf36911\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " Oct 30 23:42:40.988630 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.988553 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a7312d5-8075-4543-a074-1f176cf36911-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7312d5-8075-4543-a074-1f176cf36911-workload-socket\") pod \"3a7312d5-8075-4543-a074-1f176cf36911\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " Oct 30 23:42:40.988630 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.988587 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a7312d5-8075-4543-a074-1f176cf36911-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a7312d5-8075-4543-a074-1f176cf36911-workload-certs\") pod \"3a7312d5-8075-4543-a074-1f176cf36911\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " Oct 30 23:42:40.988729 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.988687 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a7312d5-8075-4543-a074-1f176cf36911-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a7312d5-8075-4543-a074-1f176cf36911-istio-data\") pod \"3a7312d5-8075-4543-a074-1f176cf36911\" (UID: \"3a7312d5-8075-4543-a074-1f176cf36911\") " Oct 30 23:42:40.989096 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.988717 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a7312d5-8075-4543-a074-1f176cf36911-credential-socket" pod "3a7312d5-8075-4543-a074-1f176cf36911" (UID: "3a7312d5-8075-4543-a074-1f176cf36911"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:42:40.989096 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.988903 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a7312d5-8075-4543-a074-1f176cf36911-istiod-ca-cert" pod "3a7312d5-8075-4543-a074-1f176cf36911" (UID: "3a7312d5-8075-4543-a074-1f176cf36911"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:42:40.989096 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.988936 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a7312d5-8075-4543-a074-1f176cf36911-workload-socket" pod "3a7312d5-8075-4543-a074-1f176cf36911" (UID: "3a7312d5-8075-4543-a074-1f176cf36911"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:42:40.989319 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.989211 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a7312d5-8075-4543-a074-1f176cf36911-workload-certs" pod "3a7312d5-8075-4543-a074-1f176cf36911" (UID: "3a7312d5-8075-4543-a074-1f176cf36911"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:42:40.989434 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.989414 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a7312d5-8075-4543-a074-1f176cf36911-istio-data" pod "3a7312d5-8075-4543-a074-1f176cf36911" (UID: "3a7312d5-8075-4543-a074-1f176cf36911"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:42:40.989498 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.989433 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7312d5-8075-4543-a074-1f176cf36911-credential-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:40.989498 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.989457 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a7312d5-8075-4543-a074-1f176cf36911-istiod-ca-cert\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:40.989498 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.989472 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7312d5-8075-4543-a074-1f176cf36911-workload-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:40.989498 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.989485 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a7312d5-8075-4543-a074-1f176cf36911-workload-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:40.990774 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.990750 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a7312d5-8075-4543-a074-1f176cf36911-istio-token" pod "3a7312d5-8075-4543-a074-1f176cf36911" (UID: "3a7312d5-8075-4543-a074-1f176cf36911"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:42:40.991095 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.991025 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a7312d5-8075-4543-a074-1f176cf36911-istio-podinfo" pod "3a7312d5-8075-4543-a074-1f176cf36911" (UID: "3a7312d5-8075-4543-a074-1f176cf36911"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:42:40.991095 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.991080 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a7312d5-8075-4543-a074-1f176cf36911-kube-api-access-htxb8" pod "3a7312d5-8075-4543-a074-1f176cf36911" (UID: "3a7312d5-8075-4543-a074-1f176cf36911"). InnerVolumeSpecName "kube-api-access-htxb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:42:40.991480 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:40.991461 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a7312d5-8075-4543-a074-1f176cf36911-istio-envoy" pod "3a7312d5-8075-4543-a074-1f176cf36911" (UID: "3a7312d5-8075-4543-a074-1f176cf36911"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:42:41.090341 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.090309 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a7312d5-8075-4543-a074-1f176cf36911-istio-data\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:41.090341 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.090342 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a7312d5-8075-4543-a074-1f176cf36911-istio-podinfo\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:41.090521 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.090359 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htxb8\" (UniqueName: \"kubernetes.io/projected/3a7312d5-8075-4543-a074-1f176cf36911-kube-api-access-htxb8\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:41.090521 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.090375 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a7312d5-8075-4543-a074-1f176cf36911-istio-token\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:41.090521 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.090389 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a7312d5-8075-4543-a074-1f176cf36911-istio-envoy\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:41.305774 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.305715 4952 generic.go:356] "Generic (PLEG): container finished" podID="3a7312d5-8075-4543-a074-1f176cf36911" containerID="49caa245a97c277c2ac3e396ce634547f91a811b7ea94d2039bded1c963c884b" exitCode=0 Oct 30 23:42:41.305927 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.305775 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" Oct 30 23:42:41.305927 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.305801 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" event={"ID":"3a7312d5-8075-4543-a074-1f176cf36911","Type":"ContainerDied","Data":"49caa245a97c277c2ac3e396ce634547f91a811b7ea94d2039bded1c963c884b"} Oct 30 23:42:41.305927 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.305868 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld" event={"ID":"3a7312d5-8075-4543-a074-1f176cf36911","Type":"ContainerDied","Data":"afa27661dd85d31ff5e120a73567f38653fc45fac73272ea209b8a4d65a1847d"} Oct 30 23:42:41.305927 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.305901 4952 scope.go:122] "RemoveContainer" containerID="49caa245a97c277c2ac3e396ce634547f91a811b7ea94d2039bded1c963c884b" Oct 30 23:42:41.309289 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.309266 4952 generic.go:356] "Generic (PLEG): container finished" podID="9eb85bfe-fe64-4db7-abc4-6a453d8c3060" containerID="6212e45b4f25274285b0b8bf55c23a7bb4ac32dd5f61ad03d9d361cbd54a61d8" exitCode=0 Oct 30 23:42:41.309405 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.309318 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" event={"ID":"9eb85bfe-fe64-4db7-abc4-6a453d8c3060","Type":"ContainerDied","Data":"6212e45b4f25274285b0b8bf55c23a7bb4ac32dd5f61ad03d9d361cbd54a61d8"} Oct 30 23:42:41.309405 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.309358 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" event={"ID":"9eb85bfe-fe64-4db7-abc4-6a453d8c3060","Type":"ContainerDied","Data":"80a1e028e8a5462c5430a679b9f8dd35b172220b2ca3ae68564e0fa3c1a59e11"} Oct 30 23:42:41.309522 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.309438 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w" Oct 30 23:42:41.336389 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.335954 4952 scope.go:122] "RemoveContainer" containerID="49caa245a97c277c2ac3e396ce634547f91a811b7ea94d2039bded1c963c884b" Oct 30 23:42:41.336529 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:42:41.336506 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49caa245a97c277c2ac3e396ce634547f91a811b7ea94d2039bded1c963c884b\": container with ID starting with 49caa245a97c277c2ac3e396ce634547f91a811b7ea94d2039bded1c963c884b not found: ID does not exist" containerID="49caa245a97c277c2ac3e396ce634547f91a811b7ea94d2039bded1c963c884b" Oct 30 23:42:41.336606 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.336536 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49caa245a97c277c2ac3e396ce634547f91a811b7ea94d2039bded1c963c884b"} err="failed to get container status \"49caa245a97c277c2ac3e396ce634547f91a811b7ea94d2039bded1c963c884b\": rpc error: code = NotFound desc = could not find container \"49caa245a97c277c2ac3e396ce634547f91a811b7ea94d2039bded1c963c884b\": container with ID starting with 49caa245a97c277c2ac3e396ce634547f91a811b7ea94d2039bded1c963c884b not found: ID does not exist" Oct 30 23:42:41.336606 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.336553 4952 scope.go:122] "RemoveContainer" containerID="6212e45b4f25274285b0b8bf55c23a7bb4ac32dd5f61ad03d9d361cbd54a61d8" Oct 30 23:42:41.350294 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.350274 4952 scope.go:122] "RemoveContainer" containerID="6212e45b4f25274285b0b8bf55c23a7bb4ac32dd5f61ad03d9d361cbd54a61d8" Oct 30 23:42:41.350636 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:42:41.350602 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6212e45b4f25274285b0b8bf55c23a7bb4ac32dd5f61ad03d9d361cbd54a61d8\": container with ID starting with 6212e45b4f25274285b0b8bf55c23a7bb4ac32dd5f61ad03d9d361cbd54a61d8 not found: ID does not exist" containerID="6212e45b4f25274285b0b8bf55c23a7bb4ac32dd5f61ad03d9d361cbd54a61d8" Oct 30 23:42:41.350709 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.350638 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6212e45b4f25274285b0b8bf55c23a7bb4ac32dd5f61ad03d9d361cbd54a61d8"} err="failed to get container status \"6212e45b4f25274285b0b8bf55c23a7bb4ac32dd5f61ad03d9d361cbd54a61d8\": rpc error: code = NotFound desc = could not find container \"6212e45b4f25274285b0b8bf55c23a7bb4ac32dd5f61ad03d9d361cbd54a61d8\": container with ID starting with 6212e45b4f25274285b0b8bf55c23a7bb4ac32dd5f61ad03d9d361cbd54a61d8 not found: ID does not exist" Oct 30 23:42:41.480791 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.480763 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w"] Oct 30 23:42:41.487884 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.487854 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:41.514217 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.514193 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" Oct 30 23:42:41.532145 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.532125 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-x678w"] Oct 30 23:42:41.643356 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.643280 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld"] Oct 30 23:42:41.648289 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.648254 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-sjzld"] Oct 30 23:42:41.697477 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.697454 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98027a1c-59f6-4d79-a118-dae8ed53e50b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98027a1c-59f6-4d79-a118-dae8ed53e50b-workload-socket\") pod \"98027a1c-59f6-4d79-a118-dae8ed53e50b\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " Oct 30 23:42:41.697657 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.697514 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98027a1c-59f6-4d79-a118-dae8ed53e50b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98027a1c-59f6-4d79-a118-dae8ed53e50b-istiod-ca-cert\") pod \"98027a1c-59f6-4d79-a118-dae8ed53e50b\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " Oct 30 23:42:41.697657 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.697563 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98027a1c-59f6-4d79-a118-dae8ed53e50b-istio-token\" (UniqueName: \"kubernetes.io/projected/98027a1c-59f6-4d79-a118-dae8ed53e50b-istio-token\") pod \"98027a1c-59f6-4d79-a118-dae8ed53e50b\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " Oct 30 23:42:41.697657 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.697643 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98027a1c-59f6-4d79-a118-dae8ed53e50b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98027a1c-59f6-4d79-a118-dae8ed53e50b-istio-data\") pod \"98027a1c-59f6-4d79-a118-dae8ed53e50b\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " Oct 30 23:42:41.697836 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.697707 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98027a1c-59f6-4d79-a118-dae8ed53e50b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98027a1c-59f6-4d79-a118-dae8ed53e50b-credential-socket\") pod \"98027a1c-59f6-4d79-a118-dae8ed53e50b\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " Oct 30 23:42:41.697836 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.697757 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98027a1c-59f6-4d79-a118-dae8ed53e50b-workload-socket" pod "98027a1c-59f6-4d79-a118-dae8ed53e50b" (UID: "98027a1c-59f6-4d79-a118-dae8ed53e50b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:42:41.697836 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.697764 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98027a1c-59f6-4d79-a118-dae8ed53e50b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98027a1c-59f6-4d79-a118-dae8ed53e50b-istio-envoy\") pod \"98027a1c-59f6-4d79-a118-dae8ed53e50b\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " Oct 30 23:42:41.698007 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.697840 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98027a1c-59f6-4d79-a118-dae8ed53e50b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98027a1c-59f6-4d79-a118-dae8ed53e50b-workload-certs\") pod \"98027a1c-59f6-4d79-a118-dae8ed53e50b\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " Oct 30 23:42:41.698007 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.697911 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98027a1c-59f6-4d79-a118-dae8ed53e50b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98027a1c-59f6-4d79-a118-dae8ed53e50b-istio-podinfo\") pod \"98027a1c-59f6-4d79-a118-dae8ed53e50b\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " Oct 30 23:42:41.698007 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.697954 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98027a1c-59f6-4d79-a118-dae8ed53e50b-kube-api-access-phs47\" (UniqueName: \"kubernetes.io/projected/98027a1c-59f6-4d79-a118-dae8ed53e50b-kube-api-access-phs47\") pod \"98027a1c-59f6-4d79-a118-dae8ed53e50b\" (UID: \"98027a1c-59f6-4d79-a118-dae8ed53e50b\") " Oct 30 23:42:41.698007 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.697981 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98027a1c-59f6-4d79-a118-dae8ed53e50b-istiod-ca-cert" pod "98027a1c-59f6-4d79-a118-dae8ed53e50b" (UID: "98027a1c-59f6-4d79-a118-dae8ed53e50b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:42:41.698349 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.698310 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98027a1c-59f6-4d79-a118-dae8ed53e50b-credential-socket" pod "98027a1c-59f6-4d79-a118-dae8ed53e50b" (UID: "98027a1c-59f6-4d79-a118-dae8ed53e50b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:42:41.698854 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.698828 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98027a1c-59f6-4d79-a118-dae8ed53e50b-credential-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:41.698946 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.698861 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98027a1c-59f6-4d79-a118-dae8ed53e50b-workload-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:41.698946 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.698879 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98027a1c-59f6-4d79-a118-dae8ed53e50b-istiod-ca-cert\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:41.699093 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.698962 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98027a1c-59f6-4d79-a118-dae8ed53e50b-workload-certs" pod "98027a1c-59f6-4d79-a118-dae8ed53e50b" (UID: "98027a1c-59f6-4d79-a118-dae8ed53e50b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:42:41.699093 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.699057 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98027a1c-59f6-4d79-a118-dae8ed53e50b-istio-data" pod "98027a1c-59f6-4d79-a118-dae8ed53e50b" (UID: "98027a1c-59f6-4d79-a118-dae8ed53e50b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:42:41.700300 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.700271 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98027a1c-59f6-4d79-a118-dae8ed53e50b-istio-envoy" pod "98027a1c-59f6-4d79-a118-dae8ed53e50b" (UID: "98027a1c-59f6-4d79-a118-dae8ed53e50b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:42:41.700389 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.700322 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98027a1c-59f6-4d79-a118-dae8ed53e50b-istio-token" pod "98027a1c-59f6-4d79-a118-dae8ed53e50b" (UID: "98027a1c-59f6-4d79-a118-dae8ed53e50b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:42:41.700743 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.700716 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98027a1c-59f6-4d79-a118-dae8ed53e50b-istio-podinfo" pod "98027a1c-59f6-4d79-a118-dae8ed53e50b" (UID: "98027a1c-59f6-4d79-a118-dae8ed53e50b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:42:41.700838 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.700787 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98027a1c-59f6-4d79-a118-dae8ed53e50b-kube-api-access-phs47" pod "98027a1c-59f6-4d79-a118-dae8ed53e50b" (UID: "98027a1c-59f6-4d79-a118-dae8ed53e50b"). InnerVolumeSpecName "kube-api-access-phs47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:42:41.757330 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.757306 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:41.800706 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.800673 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98027a1c-59f6-4d79-a118-dae8ed53e50b-istio-token\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:41.800857 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.800707 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98027a1c-59f6-4d79-a118-dae8ed53e50b-istio-data\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:41.800857 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.800725 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98027a1c-59f6-4d79-a118-dae8ed53e50b-istio-envoy\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:41.800857 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.800739 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98027a1c-59f6-4d79-a118-dae8ed53e50b-workload-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:41.800857 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.800753 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98027a1c-59f6-4d79-a118-dae8ed53e50b-istio-podinfo\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:41.800857 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:41.800768 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phs47\" (UniqueName: \"kubernetes.io/projected/98027a1c-59f6-4d79-a118-dae8ed53e50b-kube-api-access-phs47\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:42:42.318677 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:42.318647 4952 generic.go:356] "Generic (PLEG): container finished" podID="98027a1c-59f6-4d79-a118-dae8ed53e50b" containerID="a178f50b0fbcf4b2a7b8bf4b41a1c0b9750b7b8a9dd2a0b322ea02f8816d5c1e" exitCode=0 Oct 30 23:42:42.318869 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:42.318695 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" Oct 30 23:42:42.318869 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:42.318726 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" event={"ID":"98027a1c-59f6-4d79-a118-dae8ed53e50b","Type":"ContainerDied","Data":"a178f50b0fbcf4b2a7b8bf4b41a1c0b9750b7b8a9dd2a0b322ea02f8816d5c1e"} Oct 30 23:42:42.318869 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:42.318776 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2" event={"ID":"98027a1c-59f6-4d79-a118-dae8ed53e50b","Type":"ContainerDied","Data":"1b27845d45342de99ca52ed8d7eb5b92ee8a8b0701276c4ba2ff8fa838d4e059"} Oct 30 23:42:42.318869 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:42.318789 4952 scope.go:122] "RemoveContainer" containerID="a178f50b0fbcf4b2a7b8bf4b41a1c0b9750b7b8a9dd2a0b322ea02f8816d5c1e" Oct 30 23:42:42.330709 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:42.330692 4952 scope.go:122] "RemoveContainer" containerID="a178f50b0fbcf4b2a7b8bf4b41a1c0b9750b7b8a9dd2a0b322ea02f8816d5c1e" Oct 30 23:42:42.330974 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:42:42.330955 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a178f50b0fbcf4b2a7b8bf4b41a1c0b9750b7b8a9dd2a0b322ea02f8816d5c1e\": container with ID starting with a178f50b0fbcf4b2a7b8bf4b41a1c0b9750b7b8a9dd2a0b322ea02f8816d5c1e not found: ID does not exist" containerID="a178f50b0fbcf4b2a7b8bf4b41a1c0b9750b7b8a9dd2a0b322ea02f8816d5c1e" Oct 30 23:42:42.331115 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:42.330981 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a178f50b0fbcf4b2a7b8bf4b41a1c0b9750b7b8a9dd2a0b322ea02f8816d5c1e"} err="failed to get container status \"a178f50b0fbcf4b2a7b8bf4b41a1c0b9750b7b8a9dd2a0b322ea02f8816d5c1e\": rpc error: code = NotFound desc = could not find container \"a178f50b0fbcf4b2a7b8bf4b41a1c0b9750b7b8a9dd2a0b322ea02f8816d5c1e\": container with ID starting with a178f50b0fbcf4b2a7b8bf4b41a1c0b9750b7b8a9dd2a0b322ea02f8816d5c1e not found: ID does not exist" Oct 30 23:42:42.487290 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:42.487258 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2"] Oct 30 23:42:42.534840 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:42.534808 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-qwct2"] Oct 30 23:42:42.972839 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:42.972805 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a7312d5-8075-4543-a074-1f176cf36911" path="/var/lib/kubelet/pods/3a7312d5-8075-4543-a074-1f176cf36911/volumes" Oct 30 23:42:42.973402 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:42.973386 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98027a1c-59f6-4d79-a118-dae8ed53e50b" path="/var/lib/kubelet/pods/98027a1c-59f6-4d79-a118-dae8ed53e50b/volumes" Oct 30 23:42:42.973866 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:42.973852 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eb85bfe-fe64-4db7-abc4-6a453d8c3060" path="/var/lib/kubelet/pods/9eb85bfe-fe64-4db7-abc4-6a453d8c3060/volumes" Oct 30 23:42:43.172253 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:43.172207 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:43.405165 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:43.405133 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:45.211059 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:45.211008 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:46.448669 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:46.448631 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:46.682711 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:46.682682 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:46.909401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:46.909366 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:50.579954 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:50.579899 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:51.142960 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:42:51.142925 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:42:55.764840 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:55.764799 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:56.737499 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:56.737446 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:57.045553 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:57.045474 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:59.202574 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:59.202537 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:42:59.423547 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:42:59.423509 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:43:00.581381 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:00.581064 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:43:05.408405 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:05.408371 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:43:05.629017 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:05.628976 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:43:08.348533 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:08.348493 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:43:08.590536 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:08.590503 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:43:10.256268 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:10.256230 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:43:15.077504 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:15.077467 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:43:16.525824 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:16.525784 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:43:19.334485 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:19.334446 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:43:19.594603 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:19.594573 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:43:20.462160 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:20.462112 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:43:21.143800 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:43:21.143754 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:43:23.157007 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:23.156973 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:43:23.421715 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:23.421642 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:43:25.940792 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:25.940759 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:43:28.093584 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:28.093549 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:43:28.346448 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:28.346361 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:43:30.901776 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:30.901743 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:43:36.087664 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:36.087622 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:43:41.091900 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:41.091848 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:43:42.790604 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.790572 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w"] Oct 30 23:43:42.791194 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.791179 4952 cpu_manager.go:414] "containerMap: removing container" podUID="9eb85bfe-fe64-4db7-abc4-6a453d8c3060" containerName="istio-proxy" Oct 30 23:43:42.791256 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.791194 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eb85bfe-fe64-4db7-abc4-6a453d8c3060" containerName="istio-proxy" Oct 30 23:43:42.791256 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.791226 4952 cpu_manager.go:414] "containerMap: removing container" podUID="3a7312d5-8075-4543-a074-1f176cf36911" containerName="istio-proxy" Oct 30 23:43:42.791256 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.791231 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a7312d5-8075-4543-a074-1f176cf36911" containerName="istio-proxy" Oct 30 23:43:42.791256 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.791246 4952 cpu_manager.go:414] "containerMap: removing container" podUID="98027a1c-59f6-4d79-a118-dae8ed53e50b" containerName="istio-proxy" Oct 30 23:43:42.791256 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.791251 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98027a1c-59f6-4d79-a118-dae8ed53e50b" containerName="istio-proxy" Oct 30 23:43:42.791404 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.791266 4952 cpu_manager.go:414] "containerMap: removing container" podUID="53756ab3-37fe-4d52-b197-7066f9b257c1" containerName="istio-proxy" Oct 30 23:43:42.791404 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.791270 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53756ab3-37fe-4d52-b197-7066f9b257c1" containerName="istio-proxy" Oct 30 23:43:42.791404 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.791362 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a7312d5-8075-4543-a074-1f176cf36911" containerName="istio-proxy" Oct 30 23:43:42.791404 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.791370 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="98027a1c-59f6-4d79-a118-dae8ed53e50b" containerName="istio-proxy" Oct 30 23:43:42.791404 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.791379 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="53756ab3-37fe-4d52-b197-7066f9b257c1" containerName="istio-proxy" Oct 30 23:43:42.791404 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.791387 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eb85bfe-fe64-4db7-abc4-6a453d8c3060" containerName="istio-proxy" Oct 30 23:43:42.796322 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.796300 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" Oct 30 23:43:42.853978 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.853898 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4wk2\" (UniqueName: \"kubernetes.io/projected/d22e1d9f-31a2-43a0-999c-084069d6a9eb-kube-api-access-h4wk2\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-q466w\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" Oct 30 23:43:42.853978 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.853948 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d22e1d9f-31a2-43a0-999c-084069d6a9eb-credential-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-q466w\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" Oct 30 23:43:42.854231 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.854077 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d22e1d9f-31a2-43a0-999c-084069d6a9eb-workload-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-q466w\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" Oct 30 23:43:42.854231 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.854119 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d22e1d9f-31a2-43a0-999c-084069d6a9eb-istio-podinfo\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-q466w\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" Oct 30 23:43:42.854231 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.854156 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d22e1d9f-31a2-43a0-999c-084069d6a9eb-istiod-ca-cert\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-q466w\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" Oct 30 23:43:42.854231 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.854189 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d22e1d9f-31a2-43a0-999c-084069d6a9eb-istio-token\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-q466w\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" Oct 30 23:43:42.854397 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.854314 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d22e1d9f-31a2-43a0-999c-084069d6a9eb-workload-certs\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-q466w\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" Oct 30 23:43:42.854397 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.854354 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d22e1d9f-31a2-43a0-999c-084069d6a9eb-istio-envoy\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-q466w\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" Oct 30 23:43:42.854501 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.854481 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d22e1d9f-31a2-43a0-999c-084069d6a9eb-istio-data\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-q466w\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" Oct 30 23:43:42.867748 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.867719 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w"] Oct 30 23:43:42.955474 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.955439 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d22e1d9f-31a2-43a0-999c-084069d6a9eb-istio-token\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-q466w\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" Oct 30 23:43:42.955614 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.955480 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d22e1d9f-31a2-43a0-999c-084069d6a9eb-workload-certs\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-q466w\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" Oct 30 23:43:42.955614 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.955496 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d22e1d9f-31a2-43a0-999c-084069d6a9eb-istio-envoy\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-q466w\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" Oct 30 23:43:42.955614 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.955582 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d22e1d9f-31a2-43a0-999c-084069d6a9eb-istio-data\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-q466w\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" Oct 30 23:43:42.955726 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.955642 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4wk2\" (UniqueName: \"kubernetes.io/projected/d22e1d9f-31a2-43a0-999c-084069d6a9eb-kube-api-access-h4wk2\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-q466w\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" Oct 30 23:43:42.955726 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.955666 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d22e1d9f-31a2-43a0-999c-084069d6a9eb-credential-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-q466w\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" Oct 30 23:43:42.955726 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.955705 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d22e1d9f-31a2-43a0-999c-084069d6a9eb-workload-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-q466w\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" Oct 30 23:43:42.955873 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.955747 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d22e1d9f-31a2-43a0-999c-084069d6a9eb-istio-podinfo\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-q466w\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" Oct 30 23:43:42.955873 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.955787 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d22e1d9f-31a2-43a0-999c-084069d6a9eb-istiod-ca-cert\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-q466w\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" Oct 30 23:43:42.955986 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.955887 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d22e1d9f-31a2-43a0-999c-084069d6a9eb-workload-certs\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-q466w\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" Oct 30 23:43:42.956064 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.956017 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d22e1d9f-31a2-43a0-999c-084069d6a9eb-workload-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-q466w\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" Oct 30 23:43:42.956159 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.956134 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d22e1d9f-31a2-43a0-999c-084069d6a9eb-credential-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-q466w\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" Oct 30 23:43:42.956454 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.956435 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d22e1d9f-31a2-43a0-999c-084069d6a9eb-istiod-ca-cert\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-q466w\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" Oct 30 23:43:42.956716 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.956634 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d22e1d9f-31a2-43a0-999c-084069d6a9eb-istio-data\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-q466w\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" Oct 30 23:43:42.958075 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.958056 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d22e1d9f-31a2-43a0-999c-084069d6a9eb-istio-envoy\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-q466w\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" Oct 30 23:43:42.958400 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:42.958383 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d22e1d9f-31a2-43a0-999c-084069d6a9eb-istio-podinfo\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-q466w\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" Oct 30 23:43:43.021853 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:43.021813 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d22e1d9f-31a2-43a0-999c-084069d6a9eb-istio-token\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-q466w\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" Oct 30 23:43:43.021975 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:43.021921 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4wk2\" (UniqueName: \"kubernetes.io/projected/d22e1d9f-31a2-43a0-999c-084069d6a9eb-kube-api-access-h4wk2\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-q466w\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" Oct 30 23:43:43.110255 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:43.110231 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" Oct 30 23:43:43.553218 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:43:43.553188 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd22e1d9f_31a2_43a0_999c_084069d6a9eb.slice/crio-65fd4f9629c0a3f32e217ed18d02f6471ba069f45fd63cb7093af0109feb220d WatchSource:0}: Error finding container 65fd4f9629c0a3f32e217ed18d02f6471ba069f45fd63cb7093af0109feb220d: Status 404 returned error can't find the container with id 65fd4f9629c0a3f32e217ed18d02f6471ba069f45fd63cb7093af0109feb220d Oct 30 23:43:43.553333 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:43.553278 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w"] Oct 30 23:43:43.555780 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:43.555746 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:43:43.555865 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:43.555811 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:43:43.555865 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:43.555842 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:43:43.784404 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:43.784364 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" event={"ID":"d22e1d9f-31a2-43a0-999c-084069d6a9eb","Type":"ContainerStarted","Data":"18cc3d0fddc8a5f7c62694b0eabd1e51887acdbdcf4e8656a8d7eff92f6e6155"} Oct 30 23:43:43.784588 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:43.784426 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" event={"ID":"d22e1d9f-31a2-43a0-999c-084069d6a9eb","Type":"ContainerStarted","Data":"65fd4f9629c0a3f32e217ed18d02f6471ba069f45fd63cb7093af0109feb220d"} Oct 30 23:43:43.907334 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:43.907284 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" podStartSLOduration=1.9072699260000001 podStartE2EDuration="1.907269926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:43:43.90332685 +0000 UTC m=+2243.047157076" watchObservedRunningTime="2026-10-30 23:43:43.907269926 +0000 UTC m=+2243.051100129" Oct 30 23:43:44.111318 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:44.111265 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" Oct 30 23:43:44.113378 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:44.113344 4952 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-q466w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 23:43:44.121067 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:44.113580 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" podUID="d22e1d9f-31a2-43a0-999c-084069d6a9eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 23:43:45.111261 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:45.111226 4952 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-q466w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 23:43:45.111665 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:45.111279 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" podUID="d22e1d9f-31a2-43a0-999c-084069d6a9eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 23:43:46.111418 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:46.111383 4952 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-q466w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 23:43:46.111824 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:46.111445 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" podUID="d22e1d9f-31a2-43a0-999c-084069d6a9eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 23:43:46.123098 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:46.123030 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:43:47.110781 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:47.110743 4952 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-q466w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 23:43:47.110952 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:47.110795 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" podUID="d22e1d9f-31a2-43a0-999c-084069d6a9eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 23:43:48.111077 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:48.111016 4952 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-q466w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 23:43:48.111444 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:48.111103 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" podUID="d22e1d9f-31a2-43a0-999c-084069d6a9eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 23:43:49.111369 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:49.111338 4952 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-q466w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 23:43:49.111763 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:49.111393 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" podUID="d22e1d9f-31a2-43a0-999c-084069d6a9eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 23:43:50.115964 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:50.115934 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" Oct 30 23:43:50.116418 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:50.116391 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" Oct 30 23:43:50.117071 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:50.117024 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" Oct 30 23:43:51.141560 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:43:51.141525 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:43:51.413282 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:51.413199 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:43:56.647659 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:56.647629 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:43:57.208832 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:57.208796 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-jc5dz"] Oct 30 23:43:57.600220 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:57.600187 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:43:57.876977 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:57.876890 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:43:57.900841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:57.900813 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c"] Oct 30 23:43:57.905920 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:57.905899 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" Oct 30 23:43:57.961517 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:57.961479 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-istio-token\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-f4p5c\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" Oct 30 23:43:57.961648 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:57.961529 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-istio-podinfo\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-f4p5c\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" Oct 30 23:43:57.961648 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:57.961552 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-istio-envoy\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-f4p5c\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" Oct 30 23:43:57.961749 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:57.961675 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5hvv\" (UniqueName: \"kubernetes.io/projected/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-kube-api-access-b5hvv\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-f4p5c\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" Oct 30 23:43:57.961749 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:57.961704 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-credential-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-f4p5c\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" Oct 30 23:43:57.961749 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:57.961739 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-workload-certs\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-f4p5c\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" Oct 30 23:43:57.961885 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:57.961847 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-istio-data\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-f4p5c\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" Oct 30 23:43:57.961885 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:57.961878 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-istiod-ca-cert\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-f4p5c\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" Oct 30 23:43:57.961964 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:57.961894 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-workload-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-f4p5c\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" Oct 30 23:43:57.999400 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:57.999374 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c"] Oct 30 23:43:58.063528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:58.063490 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-istio-data\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-f4p5c\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" Oct 30 23:43:58.063687 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:58.063551 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-istiod-ca-cert\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-f4p5c\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" Oct 30 23:43:58.063687 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:58.063583 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-workload-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-f4p5c\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" Oct 30 23:43:58.063810 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:58.063776 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-istio-token\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-f4p5c\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" Oct 30 23:43:58.063876 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:58.063851 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-istio-data\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-f4p5c\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" Oct 30 23:43:58.063938 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:58.063858 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-istio-podinfo\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-f4p5c\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" Oct 30 23:43:58.063938 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:58.063908 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-istio-envoy\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-f4p5c\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" Oct 30 23:43:58.064081 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:58.063948 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-workload-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-f4p5c\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" Oct 30 23:43:58.064081 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:58.063956 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5hvv\" (UniqueName: \"kubernetes.io/projected/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-kube-api-access-b5hvv\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-f4p5c\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" Oct 30 23:43:58.064081 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:58.063993 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-credential-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-f4p5c\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" Oct 30 23:43:58.064243 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:58.064093 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-workload-certs\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-f4p5c\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" Oct 30 23:43:58.064359 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:58.064332 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-credential-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-f4p5c\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" Oct 30 23:43:58.064430 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:58.064396 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-workload-certs\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-f4p5c\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" Oct 30 23:43:58.064678 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:58.064656 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-istiod-ca-cert\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-f4p5c\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" Oct 30 23:43:58.066371 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:58.066347 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-istio-envoy\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-f4p5c\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" Oct 30 23:43:58.066552 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:58.066535 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-istio-podinfo\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-f4p5c\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" Oct 30 23:43:58.123145 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:58.123111 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-istio-token\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-f4p5c\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" Oct 30 23:43:58.123473 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:58.123449 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5hvv\" (UniqueName: \"kubernetes.io/projected/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-kube-api-access-b5hvv\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-f4p5c\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" Oct 30 23:43:58.220983 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:58.220922 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" Oct 30 23:43:58.692863 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:58.691914 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c"] Oct 30 23:43:58.692863 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:58.691980 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:43:58.692863 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:58.692088 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:43:58.692863 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:58.692133 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:43:58.923061 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:58.921528 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" event={"ID":"3e71bd07-c0e7-4013-ac2d-67206a09b3e2","Type":"ContainerStarted","Data":"9f09a8cce7ab8093e99963d1f1f7688450135ec13a984d3057ad6023723b812a"} Oct 30 23:43:58.923061 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:58.921591 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" event={"ID":"3e71bd07-c0e7-4013-ac2d-67206a09b3e2","Type":"ContainerStarted","Data":"741695f371910b49a6331c867dfc785f6c1b09c2bd52beaee7bbdb17c91df78d"} Oct 30 23:43:59.051654 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:59.051547 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" podStartSLOduration=2.051532326 podStartE2EDuration="2.051532326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:43:59.045612802 +0000 UTC m=+2258.189443008" watchObservedRunningTime="2026-10-30 23:43:59.051532326 +0000 UTC m=+2258.195362529" Oct 30 23:43:59.222107 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:59.222071 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" Oct 30 23:43:59.223426 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:59.223401 4952 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 23:43:59.223561 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:59.223454 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" podUID="3e71bd07-c0e7-4013-ac2d-67206a09b3e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 23:43:59.378070 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:43:59.378027 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:00.222373 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:00.222342 4952 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 23:44:00.222806 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:00.222402 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" podUID="3e71bd07-c0e7-4013-ac2d-67206a09b3e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 23:44:01.222013 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:01.221983 4952 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 23:44:01.222222 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:01.222118 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" podUID="3e71bd07-c0e7-4013-ac2d-67206a09b3e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 23:44:01.311842 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:01.311772 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:02.209714 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:02.209674 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-jc5dz" podUID="b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4" containerName="payload-pre-processing" containerID="cri-o://7a260691fcc10234db148b5f3584de3bc3e7491b516aa912dcb47c7b850959de" gracePeriod=25 Oct 30 23:44:02.221636 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:02.221609 4952 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 23:44:02.221766 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:02.221669 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" podUID="3e71bd07-c0e7-4013-ac2d-67206a09b3e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 23:44:02.526881 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:02.525934 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:02.685603 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:02.685248 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-jc5dz" Oct 30 23:44:02.757835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:02.757786 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4-plugins-config-volume\") pod \"b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4\" (UID: \"b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4\") " Oct 30 23:44:02.758437 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:02.758419 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4-kube-api-access-swmz7\" (UniqueName: \"kubernetes.io/projected/b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4-kube-api-access-swmz7\") pod \"b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4\" (UID: \"b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4\") " Oct 30 23:44:02.760248 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:02.760225 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4-plugins-config-volume" pod "b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4" (UID: "b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:44:02.772358 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:02.772320 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4-kube-api-access-swmz7" pod "b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4" (UID: "b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4"). InnerVolumeSpecName "kube-api-access-swmz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:44:02.869108 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:02.864943 4952 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4-plugins-config-volume\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:44:02.869108 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:02.864979 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swmz7\" (UniqueName: \"kubernetes.io/projected/b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4-kube-api-access-swmz7\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:44:02.891258 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:02.890443 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:02.953630 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:02.953318 4952 generic.go:356] "Generic (PLEG): container finished" podID="b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4" containerID="7a260691fcc10234db148b5f3584de3bc3e7491b516aa912dcb47c7b850959de" exitCode=0 Oct 30 23:44:02.953630 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:02.953374 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-jc5dz" Oct 30 23:44:02.953630 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:02.953412 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-jc5dz" event={"ID":"b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4","Type":"ContainerDied","Data":"7a260691fcc10234db148b5f3584de3bc3e7491b516aa912dcb47c7b850959de"} Oct 30 23:44:02.953630 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:02.953452 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-jc5dz" event={"ID":"b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4","Type":"ContainerDied","Data":"e0e534f3ae0cf97d5cc723dc5e8e2c98be0a8a1479a7477828a81f6ab9a32a85"} Oct 30 23:44:02.953630 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:02.953472 4952 scope.go:122] "RemoveContainer" containerID="7a260691fcc10234db148b5f3584de3bc3e7491b516aa912dcb47c7b850959de" Oct 30 23:44:02.999249 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:02.982837 4952 scope.go:122] "RemoveContainer" containerID="7a260691fcc10234db148b5f3584de3bc3e7491b516aa912dcb47c7b850959de" Oct 30 23:44:02.999249 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:44:02.997155 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a260691fcc10234db148b5f3584de3bc3e7491b516aa912dcb47c7b850959de\": container with ID starting with 7a260691fcc10234db148b5f3584de3bc3e7491b516aa912dcb47c7b850959de not found: ID does not exist" containerID="7a260691fcc10234db148b5f3584de3bc3e7491b516aa912dcb47c7b850959de" Oct 30 23:44:02.999249 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:02.997221 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a260691fcc10234db148b5f3584de3bc3e7491b516aa912dcb47c7b850959de"} err="failed to get container status \"7a260691fcc10234db148b5f3584de3bc3e7491b516aa912dcb47c7b850959de\": rpc error: code = NotFound desc = could not find container \"7a260691fcc10234db148b5f3584de3bc3e7491b516aa912dcb47c7b850959de\": container with ID starting with 7a260691fcc10234db148b5f3584de3bc3e7491b516aa912dcb47c7b850959de not found: ID does not exist" Oct 30 23:44:03.119683 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:03.119641 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-jc5dz"] Oct 30 23:44:03.126724 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:03.126699 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-jc5dz"] Oct 30 23:44:03.222319 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:03.222261 4952 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 23:44:03.222482 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:03.222357 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" podUID="3e71bd07-c0e7-4013-ac2d-67206a09b3e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 23:44:03.236337 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:03.236309 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:03.943742 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:03.943706 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:04.222316 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:04.222223 4952 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 23:44:04.222316 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:04.222294 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" podUID="3e71bd07-c0e7-4013-ac2d-67206a09b3e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 23:44:04.288656 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:04.288428 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:04.608472 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:04.608427 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:04.862708 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:04.862634 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:04.971187 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:04.971149 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4" path="/var/lib/kubelet/pods/b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4/volumes" Oct 30 23:44:05.304604 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:05.304519 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:05.383805 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:05.383774 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" Oct 30 23:44:05.384177 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:05.384151 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" Oct 30 23:44:05.384925 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:05.384908 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" Oct 30 23:44:09.860945 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:09.860897 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:10.127845 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:10.127758 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:10.390742 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:10.390664 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:12.307772 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:12.307743 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:12.573847 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:12.573719 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:12.837052 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:12.837014 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:13.544131 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:13.544031 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:14.186654 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:14.186615 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:14.410339 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:14.410306 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6"] Oct 30 23:44:14.410563 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:14.410543 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6" podUID="88e27e01-b433-41c6-b19c-657bcb771e1c" containerName="maas-api" containerID="cri-o://7d62cde5dd23b8094334e3708fdf4c3589182be864b439c9c4437f5981399029" gracePeriod=30 Oct 30 23:44:14.560861 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:14.560823 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:14.757285 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:14.757259 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6" Oct 30 23:44:14.784689 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:14.784627 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88e27e01-b433-41c6-b19c-657bcb771e1c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/88e27e01-b433-41c6-b19c-657bcb771e1c-metrics-certs\") pod \"88e27e01-b433-41c6-b19c-657bcb771e1c\" (UID: \"88e27e01-b433-41c6-b19c-657bcb771e1c\") " Oct 30 23:44:14.784828 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:14.784760 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88e27e01-b433-41c6-b19c-657bcb771e1c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88e27e01-b433-41c6-b19c-657bcb771e1c-maas-api-tls\") pod \"88e27e01-b433-41c6-b19c-657bcb771e1c\" (UID: \"88e27e01-b433-41c6-b19c-657bcb771e1c\") " Oct 30 23:44:14.784828 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:14.784801 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88e27e01-b433-41c6-b19c-657bcb771e1c-kube-api-access-j9v6t\" (UniqueName: \"kubernetes.io/projected/88e27e01-b433-41c6-b19c-657bcb771e1c-kube-api-access-j9v6t\") pod \"88e27e01-b433-41c6-b19c-657bcb771e1c\" (UID: \"88e27e01-b433-41c6-b19c-657bcb771e1c\") " Oct 30 23:44:14.784945 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:14.784854 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88e27e01-b433-41c6-b19c-657bcb771e1c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e27e01-b433-41c6-b19c-657bcb771e1c-openshift-service-ca\") pod \"88e27e01-b433-41c6-b19c-657bcb771e1c\" (UID: \"88e27e01-b433-41c6-b19c-657bcb771e1c\") " Oct 30 23:44:14.785537 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:14.785336 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e27e01-b433-41c6-b19c-657bcb771e1c-openshift-service-ca" pod "88e27e01-b433-41c6-b19c-657bcb771e1c" (UID: "88e27e01-b433-41c6-b19c-657bcb771e1c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:44:14.785537 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:14.785498 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e27e01-b433-41c6-b19c-657bcb771e1c-openshift-service-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:44:14.787224 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:14.787196 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88e27e01-b433-41c6-b19c-657bcb771e1c-maas-api-tls" pod "88e27e01-b433-41c6-b19c-657bcb771e1c" (UID: "88e27e01-b433-41c6-b19c-657bcb771e1c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:44:14.787653 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:14.787627 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88e27e01-b433-41c6-b19c-657bcb771e1c-metrics-certs" pod "88e27e01-b433-41c6-b19c-657bcb771e1c" (UID: "88e27e01-b433-41c6-b19c-657bcb771e1c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:44:14.787931 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:14.787913 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e27e01-b433-41c6-b19c-657bcb771e1c-kube-api-access-j9v6t" pod "88e27e01-b433-41c6-b19c-657bcb771e1c" (UID: "88e27e01-b433-41c6-b19c-657bcb771e1c"). InnerVolumeSpecName "kube-api-access-j9v6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:44:14.865010 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:14.864974 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:14.886962 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:14.886923 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88e27e01-b433-41c6-b19c-657bcb771e1c-metrics-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:44:14.887217 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:14.887198 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88e27e01-b433-41c6-b19c-657bcb771e1c-maas-api-tls\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:44:14.887327 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:14.887313 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9v6t\" (UniqueName: \"kubernetes.io/projected/88e27e01-b433-41c6-b19c-657bcb771e1c-kube-api-access-j9v6t\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:44:15.128653 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:15.128622 4952 generic.go:356] "Generic (PLEG): container finished" podID="88e27e01-b433-41c6-b19c-657bcb771e1c" containerID="7d62cde5dd23b8094334e3708fdf4c3589182be864b439c9c4437f5981399029" exitCode=0 Oct 30 23:44:15.128816 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:15.128677 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6" Oct 30 23:44:15.128816 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:15.128695 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6" event={"ID":"88e27e01-b433-41c6-b19c-657bcb771e1c","Type":"ContainerDied","Data":"7d62cde5dd23b8094334e3708fdf4c3589182be864b439c9c4437f5981399029"} Oct 30 23:44:15.128816 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:15.128748 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6" event={"ID":"88e27e01-b433-41c6-b19c-657bcb771e1c","Type":"ContainerDied","Data":"ad8e675b6375bfaddae9e684dd304428845bea544339e0e1ab459030aaa82f56"} Oct 30 23:44:15.128816 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:15.128776 4952 scope.go:122] "RemoveContainer" containerID="7d62cde5dd23b8094334e3708fdf4c3589182be864b439c9c4437f5981399029" Oct 30 23:44:15.140524 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:15.140504 4952 scope.go:122] "RemoveContainer" containerID="7d62cde5dd23b8094334e3708fdf4c3589182be864b439c9c4437f5981399029" Oct 30 23:44:15.140828 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:44:15.140806 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d62cde5dd23b8094334e3708fdf4c3589182be864b439c9c4437f5981399029\": container with ID starting with 7d62cde5dd23b8094334e3708fdf4c3589182be864b439c9c4437f5981399029 not found: ID does not exist" containerID="7d62cde5dd23b8094334e3708fdf4c3589182be864b439c9c4437f5981399029" Oct 30 23:44:15.140899 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:15.140837 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d62cde5dd23b8094334e3708fdf4c3589182be864b439c9c4437f5981399029"} err="failed to get container status \"7d62cde5dd23b8094334e3708fdf4c3589182be864b439c9c4437f5981399029\": rpc error: code = NotFound desc = could not find container \"7d62cde5dd23b8094334e3708fdf4c3589182be864b439c9c4437f5981399029\": container with ID starting with 7d62cde5dd23b8094334e3708fdf4c3589182be864b439c9c4437f5981399029 not found: ID does not exist" Oct 30 23:44:15.298832 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:15.298804 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6"] Oct 30 23:44:15.350755 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:15.350724 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7d1536-7dd66fddd8-q72p6"] Oct 30 23:44:16.362755 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:16.362724 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:16.651634 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:16.651539 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:16.914204 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:16.914117 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:16.972144 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:16.972118 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88e27e01-b433-41c6-b19c-657bcb771e1c" path="/var/lib/kubelet/pods/88e27e01-b433-41c6-b19c-657bcb771e1c/volumes" Oct 30 23:44:17.350422 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:17.350392 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:18.594269 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:18.594232 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:18.844756 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:18.844676 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:21.142238 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:44:21.142206 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:44:21.189731 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:21.189691 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v"] Oct 30 23:44:21.189954 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:21.189930 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v" podUID="bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04" containerName="maas-api" containerID="cri-o://baab6798f685b58b9629c815361cc82e4b4aca4b8b30f298d467ba535496b3f7" gracePeriod=30 Oct 30 23:44:21.383931 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:21.383894 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:21.521155 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:21.521121 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v" Oct 30 23:44:21.590488 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:21.590449 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04-kube-api-access-z8rkb\" (UniqueName: \"kubernetes.io/projected/bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04-kube-api-access-z8rkb\") pod \"bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04\" (UID: \"bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04\") " Oct 30 23:44:21.590777 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:21.590753 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04-openshift-service-ca\") pod \"bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04\" (UID: \"bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04\") " Oct 30 23:44:21.591410 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:21.591380 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04-openshift-service-ca" pod "bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04" (UID: "bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:44:21.592858 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:21.592009 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04-maas-api-tls\") pod \"bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04\" (UID: \"bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04\") " Oct 30 23:44:21.596367 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:21.596343 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04-metrics-certs\") pod \"bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04\" (UID: \"bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04\") " Oct 30 23:44:21.596962 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:21.596616 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04-kube-api-access-z8rkb" pod "bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04" (UID: "bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04"). InnerVolumeSpecName "kube-api-access-z8rkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:44:21.597776 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:21.597750 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8rkb\" (UniqueName: \"kubernetes.io/projected/bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04-kube-api-access-z8rkb\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:44:21.597913 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:21.597898 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04-openshift-service-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:44:21.599744 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:21.598818 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04-maas-api-tls" pod "bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04" (UID: "bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:44:21.607235 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:21.607198 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04-metrics-certs" pod "bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04" (UID: "bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:44:21.699532 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:21.699497 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04-maas-api-tls\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:44:21.699758 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:21.699739 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04-metrics-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:44:21.914966 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:21.914933 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:22.136561 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:22.136530 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:22.180293 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:22.180205 4952 generic.go:356] "Generic (PLEG): container finished" podID="bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04" containerID="baab6798f685b58b9629c815361cc82e4b4aca4b8b30f298d467ba535496b3f7" exitCode=0 Oct 30 23:44:22.180786 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:22.180285 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v" Oct 30 23:44:22.180786 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:22.180289 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v" event={"ID":"bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04","Type":"ContainerDied","Data":"baab6798f685b58b9629c815361cc82e4b4aca4b8b30f298d467ba535496b3f7"} Oct 30 23:44:22.180786 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:22.180435 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v" event={"ID":"bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04","Type":"ContainerDied","Data":"aef6cbcc30b05fa90590c6cc3431da7bfeb16bf882d5ac1e1478a0fa4f0e4878"} Oct 30 23:44:22.180786 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:22.180460 4952 scope.go:122] "RemoveContainer" containerID="baab6798f685b58b9629c815361cc82e4b4aca4b8b30f298d467ba535496b3f7" Oct 30 23:44:22.192492 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:22.192343 4952 scope.go:122] "RemoveContainer" containerID="baab6798f685b58b9629c815361cc82e4b4aca4b8b30f298d467ba535496b3f7" Oct 30 23:44:22.192601 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:44:22.192581 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baab6798f685b58b9629c815361cc82e4b4aca4b8b30f298d467ba535496b3f7\": container with ID starting with baab6798f685b58b9629c815361cc82e4b4aca4b8b30f298d467ba535496b3f7 not found: ID does not exist" containerID="baab6798f685b58b9629c815361cc82e4b4aca4b8b30f298d467ba535496b3f7" Oct 30 23:44:22.192642 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:22.192611 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baab6798f685b58b9629c815361cc82e4b4aca4b8b30f298d467ba535496b3f7"} err="failed to get container status \"baab6798f685b58b9629c815361cc82e4b4aca4b8b30f298d467ba535496b3f7\": rpc error: code = NotFound desc = could not find container \"baab6798f685b58b9629c815361cc82e4b4aca4b8b30f298d467ba535496b3f7\": container with ID starting with baab6798f685b58b9629c815361cc82e4b4aca4b8b30f298d467ba535496b3f7 not found: ID does not exist" Oct 30 23:44:22.356777 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:22.353093 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v"] Oct 30 23:44:22.362059 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:22.357383 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0a7530-67c6f754d9-s9l4v"] Oct 30 23:44:22.613392 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:22.613342 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:22.972298 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:22.972227 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04" path="/var/lib/kubelet/pods/bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04/volumes" Oct 30 23:44:22.984773 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:22.984737 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:23.207828 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:23.207798 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:23.517727 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:23.517690 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:23.743240 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:23.743212 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:23.847162 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:23.847134 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk"] Oct 30 23:44:23.847380 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:23.847355 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk" podUID="b155a013-f16a-481e-a4f0-a1afd8140a9f" containerName="maas-api" containerID="cri-o://61f01fc60b51dc83ecd71ed87e7267e659cd877a8c225c3d4e18eb74b6052764" gracePeriod=30 Oct 30 23:44:23.975098 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:23.975018 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:24.182852 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:24.182819 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk" Oct 30 23:44:24.198080 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:24.198015 4952 generic.go:356] "Generic (PLEG): container finished" podID="b155a013-f16a-481e-a4f0-a1afd8140a9f" containerID="61f01fc60b51dc83ecd71ed87e7267e659cd877a8c225c3d4e18eb74b6052764" exitCode=0 Oct 30 23:44:24.198080 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:24.198069 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk" event={"ID":"b155a013-f16a-481e-a4f0-a1afd8140a9f","Type":"ContainerDied","Data":"61f01fc60b51dc83ecd71ed87e7267e659cd877a8c225c3d4e18eb74b6052764"} Oct 30 23:44:24.198301 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:24.198137 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk" event={"ID":"b155a013-f16a-481e-a4f0-a1afd8140a9f","Type":"ContainerDied","Data":"26e8d7bb1dc501e4fe358066cd317effcb71e9f7a25aa7996d2a51596111eeee"} Oct 30 23:44:24.198301 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:24.198080 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk" Oct 30 23:44:24.198301 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:24.198168 4952 scope.go:122] "RemoveContainer" containerID="61f01fc60b51dc83ecd71ed87e7267e659cd877a8c225c3d4e18eb74b6052764" Oct 30 23:44:24.208814 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:24.208795 4952 scope.go:122] "RemoveContainer" containerID="61f01fc60b51dc83ecd71ed87e7267e659cd877a8c225c3d4e18eb74b6052764" Oct 30 23:44:24.209187 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:44:24.209128 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61f01fc60b51dc83ecd71ed87e7267e659cd877a8c225c3d4e18eb74b6052764\": container with ID starting with 61f01fc60b51dc83ecd71ed87e7267e659cd877a8c225c3d4e18eb74b6052764 not found: ID does not exist" containerID="61f01fc60b51dc83ecd71ed87e7267e659cd877a8c225c3d4e18eb74b6052764" Oct 30 23:44:24.209187 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:24.209170 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61f01fc60b51dc83ecd71ed87e7267e659cd877a8c225c3d4e18eb74b6052764"} err="failed to get container status \"61f01fc60b51dc83ecd71ed87e7267e659cd877a8c225c3d4e18eb74b6052764\": rpc error: code = NotFound desc = could not find container \"61f01fc60b51dc83ecd71ed87e7267e659cd877a8c225c3d4e18eb74b6052764\": container with ID starting with 61f01fc60b51dc83ecd71ed87e7267e659cd877a8c225c3d4e18eb74b6052764 not found: ID does not exist" Oct 30 23:44:24.213691 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:24.213672 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:24.249358 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:24.249327 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b155a013-f16a-481e-a4f0-a1afd8140a9f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b155a013-f16a-481e-a4f0-a1afd8140a9f-openshift-service-ca\") pod \"b155a013-f16a-481e-a4f0-a1afd8140a9f\" (UID: \"b155a013-f16a-481e-a4f0-a1afd8140a9f\") " Oct 30 23:44:24.249488 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:24.249389 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b155a013-f16a-481e-a4f0-a1afd8140a9f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b155a013-f16a-481e-a4f0-a1afd8140a9f-metrics-certs\") pod \"b155a013-f16a-481e-a4f0-a1afd8140a9f\" (UID: \"b155a013-f16a-481e-a4f0-a1afd8140a9f\") " Oct 30 23:44:24.249488 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:24.249429 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b155a013-f16a-481e-a4f0-a1afd8140a9f-kube-api-access-qnvwj\" (UniqueName: \"kubernetes.io/projected/b155a013-f16a-481e-a4f0-a1afd8140a9f-kube-api-access-qnvwj\") pod \"b155a013-f16a-481e-a4f0-a1afd8140a9f\" (UID: \"b155a013-f16a-481e-a4f0-a1afd8140a9f\") " Oct 30 23:44:24.249614 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:24.249531 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b155a013-f16a-481e-a4f0-a1afd8140a9f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b155a013-f16a-481e-a4f0-a1afd8140a9f-maas-api-tls\") pod \"b155a013-f16a-481e-a4f0-a1afd8140a9f\" (UID: \"b155a013-f16a-481e-a4f0-a1afd8140a9f\") " Oct 30 23:44:24.249761 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:24.249736 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b155a013-f16a-481e-a4f0-a1afd8140a9f-openshift-service-ca" pod "b155a013-f16a-481e-a4f0-a1afd8140a9f" (UID: "b155a013-f16a-481e-a4f0-a1afd8140a9f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:44:24.250227 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:24.250185 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b155a013-f16a-481e-a4f0-a1afd8140a9f-openshift-service-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:44:24.251837 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:24.251810 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b155a013-f16a-481e-a4f0-a1afd8140a9f-metrics-certs" pod "b155a013-f16a-481e-a4f0-a1afd8140a9f" (UID: "b155a013-f16a-481e-a4f0-a1afd8140a9f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:44:24.252294 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:24.252269 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b155a013-f16a-481e-a4f0-a1afd8140a9f-maas-api-tls" pod "b155a013-f16a-481e-a4f0-a1afd8140a9f" (UID: "b155a013-f16a-481e-a4f0-a1afd8140a9f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:44:24.252385 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:24.252293 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b155a013-f16a-481e-a4f0-a1afd8140a9f-kube-api-access-qnvwj" pod "b155a013-f16a-481e-a4f0-a1afd8140a9f" (UID: "b155a013-f16a-481e-a4f0-a1afd8140a9f"). InnerVolumeSpecName "kube-api-access-qnvwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:44:24.353450 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:24.353414 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b155a013-f16a-481e-a4f0-a1afd8140a9f-maas-api-tls\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:44:24.353733 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:24.353679 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b155a013-f16a-481e-a4f0-a1afd8140a9f-metrics-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:44:24.353733 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:24.353706 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnvwj\" (UniqueName: \"kubernetes.io/projected/b155a013-f16a-481e-a4f0-a1afd8140a9f-kube-api-access-qnvwj\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:44:24.622430 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:24.622391 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk"] Oct 30 23:44:24.632121 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:24.632094 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ba13e7-5c9b6698bc-jvssk"] Oct 30 23:44:24.671439 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:24.671364 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:24.933403 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:24.933335 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:24.974433 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:24.974394 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b155a013-f16a-481e-a4f0-a1afd8140a9f" path="/var/lib/kubelet/pods/b155a013-f16a-481e-a4f0-a1afd8140a9f/volumes" Oct 30 23:44:25.162696 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:25.162625 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:25.732669 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:25.732628 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:26.014594 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:26.014419 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:26.290672 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:26.290565 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:26.647917 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:26.647882 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:26.867696 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:26.867666 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:29.852335 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:29.852302 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:30.109175 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:30.108977 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:30.332163 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:30.332133 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:30.890240 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:30.890205 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:31.198010 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:31.197903 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:31.421250 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:31.421214 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:31.696349 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:31.696316 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw"] Oct 30 23:44:31.835067 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:31.835001 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:32.059721 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:32.059645 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:32.370465 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:32.370437 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:32.988792 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:32.988706 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:33.224652 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:33.224613 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:33.652714 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:33.652672 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:33.881317 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:33.881283 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:34.104536 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:34.104501 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:34.335487 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:34.335449 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:34.788390 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:34.788352 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:35.458114 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:35.458030 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:36.026909 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:36.026875 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:36.496615 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:36.496583 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:44:36.645955 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:36.645918 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:37.349994 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:37.349951 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:37.698625 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:37.698537 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:37.969478 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:37.969392 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:38.213898 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:38.213840 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:39.211100 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:39.210995 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:39.444169 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:39.444120 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:39.720006 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:39.719965 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:39.957392 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:39.957349 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:40.189256 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:40.189222 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:40.515957 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:40.515886 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:40.748347 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:40.748305 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:41.081996 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:41.081967 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:42.281407 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:42.281366 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:42.760947 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:42.760908 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:42.988894 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:42.988832 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:45.273401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:45.273340 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:45.490967 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:45.490924 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz"] Oct 30 23:44:45.510765 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:45.510731 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:45.539222 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:45.539129 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg"] Oct 30 23:44:45.544237 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:45.544203 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll"] Oct 30 23:44:45.756090 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:45.754954 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:46.719118 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:46.719083 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" podUID="86ec338e-ad13-4029-a1cc-88650c6a27ca" containerName="main" containerID="cri-o://4093eb89cc76ec4aa91987653e2497364397fbf33fe1283aef0cf97ecb2d202d" gracePeriod=45 Oct 30 23:44:46.837170 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:46.837140 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" podUID="86ec338e-ad13-4029-a1cc-88650c6a27ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": dial tcp 10.132.0.67:8000: connect: connection refused" Oct 30 23:44:47.050076 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:47.050024 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" Oct 30 23:44:47.151160 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:47.151131 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86ec338e-ad13-4029-a1cc-88650c6a27ca-tls-certs\" (UniqueName: \"kubernetes.io/secret/86ec338e-ad13-4029-a1cc-88650c6a27ca-tls-certs\") pod \"86ec338e-ad13-4029-a1cc-88650c6a27ca\" (UID: \"86ec338e-ad13-4029-a1cc-88650c6a27ca\") " Oct 30 23:44:47.151349 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:47.151222 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ec338e-ad13-4029-a1cc-88650c6a27ca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/86ec338e-ad13-4029-a1cc-88650c6a27ca-dshm\") pod \"86ec338e-ad13-4029-a1cc-88650c6a27ca\" (UID: \"86ec338e-ad13-4029-a1cc-88650c6a27ca\") " Oct 30 23:44:47.151349 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:47.151328 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ec338e-ad13-4029-a1cc-88650c6a27ca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86ec338e-ad13-4029-a1cc-88650c6a27ca-tmp-dir\") pod \"86ec338e-ad13-4029-a1cc-88650c6a27ca\" (UID: \"86ec338e-ad13-4029-a1cc-88650c6a27ca\") " Oct 30 23:44:47.151517 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:47.151492 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ec338e-ad13-4029-a1cc-88650c6a27ca-home\" (UniqueName: \"kubernetes.io/empty-dir/86ec338e-ad13-4029-a1cc-88650c6a27ca-home\") pod \"86ec338e-ad13-4029-a1cc-88650c6a27ca\" (UID: \"86ec338e-ad13-4029-a1cc-88650c6a27ca\") " Oct 30 23:44:47.151652 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:47.151554 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86ec338e-ad13-4029-a1cc-88650c6a27ca-kube-api-access-2phtn\" (UniqueName: \"kubernetes.io/projected/86ec338e-ad13-4029-a1cc-88650c6a27ca-kube-api-access-2phtn\") pod \"86ec338e-ad13-4029-a1cc-88650c6a27ca\" (UID: \"86ec338e-ad13-4029-a1cc-88650c6a27ca\") " Oct 30 23:44:47.151652 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:47.151586 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ec338e-ad13-4029-a1cc-88650c6a27ca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86ec338e-ad13-4029-a1cc-88650c6a27ca-model-cache\") pod \"86ec338e-ad13-4029-a1cc-88650c6a27ca\" (UID: \"86ec338e-ad13-4029-a1cc-88650c6a27ca\") " Oct 30 23:44:47.151652 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:47.151584 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ec338e-ad13-4029-a1cc-88650c6a27ca-tmp-dir" pod "86ec338e-ad13-4029-a1cc-88650c6a27ca" (UID: "86ec338e-ad13-4029-a1cc-88650c6a27ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:44:47.151816 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:47.151679 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ec338e-ad13-4029-a1cc-88650c6a27ca-home" pod "86ec338e-ad13-4029-a1cc-88650c6a27ca" (UID: "86ec338e-ad13-4029-a1cc-88650c6a27ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:44:47.151949 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:47.151927 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ec338e-ad13-4029-a1cc-88650c6a27ca-model-cache" pod "86ec338e-ad13-4029-a1cc-88650c6a27ca" (UID: "86ec338e-ad13-4029-a1cc-88650c6a27ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:44:47.152517 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:47.152497 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86ec338e-ad13-4029-a1cc-88650c6a27ca-model-cache\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:44:47.152613 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:47.152526 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86ec338e-ad13-4029-a1cc-88650c6a27ca-tmp-dir\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:44:47.152613 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:47.152543 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86ec338e-ad13-4029-a1cc-88650c6a27ca-home\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:44:47.153892 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:47.153864 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ec338e-ad13-4029-a1cc-88650c6a27ca-dshm" pod "86ec338e-ad13-4029-a1cc-88650c6a27ca" (UID: "86ec338e-ad13-4029-a1cc-88650c6a27ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:44:47.153984 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:47.153898 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ec338e-ad13-4029-a1cc-88650c6a27ca-kube-api-access-2phtn" pod "86ec338e-ad13-4029-a1cc-88650c6a27ca" (UID: "86ec338e-ad13-4029-a1cc-88650c6a27ca"). InnerVolumeSpecName "kube-api-access-2phtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:44:47.153984 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:47.153918 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86ec338e-ad13-4029-a1cc-88650c6a27ca-tls-certs" pod "86ec338e-ad13-4029-a1cc-88650c6a27ca" (UID: "86ec338e-ad13-4029-a1cc-88650c6a27ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:44:47.253568 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:47.253537 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2phtn\" (UniqueName: \"kubernetes.io/projected/86ec338e-ad13-4029-a1cc-88650c6a27ca-kube-api-access-2phtn\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:44:47.253568 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:47.253566 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86ec338e-ad13-4029-a1cc-88650c6a27ca-tls-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:44:47.253568 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:47.253576 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86ec338e-ad13-4029-a1cc-88650c6a27ca-dshm\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:44:47.459347 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:47.459311 4952 generic.go:356] "Generic (PLEG): container finished" podID="86ec338e-ad13-4029-a1cc-88650c6a27ca" containerID="4093eb89cc76ec4aa91987653e2497364397fbf33fe1283aef0cf97ecb2d202d" exitCode=0 Oct 30 23:44:47.459560 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:47.459368 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" Oct 30 23:44:47.459560 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:47.459416 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" event={"ID":"86ec338e-ad13-4029-a1cc-88650c6a27ca","Type":"ContainerDied","Data":"4093eb89cc76ec4aa91987653e2497364397fbf33fe1283aef0cf97ecb2d202d"} Oct 30 23:44:47.459560 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:47.459486 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw" event={"ID":"86ec338e-ad13-4029-a1cc-88650c6a27ca","Type":"ContainerDied","Data":"b7c6ec0666ab4ab3ad12c8971d0b77c96e4a16a2e0c3022fd524bf4200e1829c"} Oct 30 23:44:47.459560 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:47.459518 4952 scope.go:122] "RemoveContainer" containerID="4093eb89cc76ec4aa91987653e2497364397fbf33fe1283aef0cf97ecb2d202d" Oct 30 23:44:47.475425 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:47.475193 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:47.478449 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:47.478428 4952 scope.go:122] "RemoveContainer" containerID="4093eb89cc76ec4aa91987653e2497364397fbf33fe1283aef0cf97ecb2d202d" Oct 30 23:44:47.478794 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:44:47.478763 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4093eb89cc76ec4aa91987653e2497364397fbf33fe1283aef0cf97ecb2d202d\": container with ID starting with 4093eb89cc76ec4aa91987653e2497364397fbf33fe1283aef0cf97ecb2d202d not found: ID does not exist" containerID="4093eb89cc76ec4aa91987653e2497364397fbf33fe1283aef0cf97ecb2d202d" Oct 30 23:44:47.478904 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:47.478819 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4093eb89cc76ec4aa91987653e2497364397fbf33fe1283aef0cf97ecb2d202d"} err="failed to get container status \"4093eb89cc76ec4aa91987653e2497364397fbf33fe1283aef0cf97ecb2d202d\": rpc error: code = NotFound desc = could not find container \"4093eb89cc76ec4aa91987653e2497364397fbf33fe1283aef0cf97ecb2d202d\": container with ID starting with 4093eb89cc76ec4aa91987653e2497364397fbf33fe1283aef0cf97ecb2d202d not found: ID does not exist" Oct 30 23:44:47.624133 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:47.624096 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw"] Oct 30 23:44:47.627502 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:47.627480 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-057303/simulator-w0-057303-kserve-66b7c9b57f-wt5qw"] Oct 30 23:44:48.971203 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:48.971172 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86ec338e-ad13-4029-a1cc-88650c6a27ca" path="/var/lib/kubelet/pods/86ec338e-ad13-4029-a1cc-88650c6a27ca/volumes" Oct 30 23:44:51.143351 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:44:51.143316 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:44:52.757462 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:52.757429 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:57.304455 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:57.304409 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:57.441180 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:57.441141 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-55666687-7659595c45-jwcgn"] Oct 30 23:44:57.441483 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:57.441450 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55666687-7659595c45-jwcgn" podUID="8f0f494d-1bc3-4981-8203-4adb0d1b0340" containerName="maas-api" containerID="cri-o://cd44f411eba882c4bc1f492dd926c35d21629a4cdfebe402105dc7699b8d2c9e" gracePeriod=30 Oct 30 23:44:57.524132 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:57.524102 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:57.738786 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:57.738744 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:57.797174 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:57.797148 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55666687-7659595c45-jwcgn" Oct 30 23:44:57.897063 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:57.892875 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f0f494d-1bc3-4981-8203-4adb0d1b0340-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f0f494d-1bc3-4981-8203-4adb0d1b0340-openshift-service-ca\") pod \"8f0f494d-1bc3-4981-8203-4adb0d1b0340\" (UID: \"8f0f494d-1bc3-4981-8203-4adb0d1b0340\") " Oct 30 23:44:57.897063 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:57.892968 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f0f494d-1bc3-4981-8203-4adb0d1b0340-kube-api-access-fzfsq\" (UniqueName: \"kubernetes.io/projected/8f0f494d-1bc3-4981-8203-4adb0d1b0340-kube-api-access-fzfsq\") pod \"8f0f494d-1bc3-4981-8203-4adb0d1b0340\" (UID: \"8f0f494d-1bc3-4981-8203-4adb0d1b0340\") " Oct 30 23:44:57.897063 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:57.893098 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f0f494d-1bc3-4981-8203-4adb0d1b0340-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f0f494d-1bc3-4981-8203-4adb0d1b0340-maas-api-tls\") pod \"8f0f494d-1bc3-4981-8203-4adb0d1b0340\" (UID: \"8f0f494d-1bc3-4981-8203-4adb0d1b0340\") " Oct 30 23:44:57.897063 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:57.893204 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f0f494d-1bc3-4981-8203-4adb0d1b0340-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f0f494d-1bc3-4981-8203-4adb0d1b0340-metrics-certs\") pod \"8f0f494d-1bc3-4981-8203-4adb0d1b0340\" (UID: \"8f0f494d-1bc3-4981-8203-4adb0d1b0340\") " Oct 30 23:44:57.905433 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:57.901518 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f0f494d-1bc3-4981-8203-4adb0d1b0340-openshift-service-ca" pod "8f0f494d-1bc3-4981-8203-4adb0d1b0340" (UID: "8f0f494d-1bc3-4981-8203-4adb0d1b0340"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:44:57.905433 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:57.902439 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f0f494d-1bc3-4981-8203-4adb0d1b0340-metrics-certs" pod "8f0f494d-1bc3-4981-8203-4adb0d1b0340" (UID: "8f0f494d-1bc3-4981-8203-4adb0d1b0340"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:44:57.960052 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:57.959987 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f0f494d-1bc3-4981-8203-4adb0d1b0340-maas-api-tls" pod "8f0f494d-1bc3-4981-8203-4adb0d1b0340" (UID: "8f0f494d-1bc3-4981-8203-4adb0d1b0340"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:44:57.986577 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:57.984803 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f0f494d-1bc3-4981-8203-4adb0d1b0340-kube-api-access-fzfsq" pod "8f0f494d-1bc3-4981-8203-4adb0d1b0340" (UID: "8f0f494d-1bc3-4981-8203-4adb0d1b0340"). InnerVolumeSpecName "kube-api-access-fzfsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:44:57.995634 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:57.995591 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f0f494d-1bc3-4981-8203-4adb0d1b0340-maas-api-tls\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:44:57.995634 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:57.995629 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f0f494d-1bc3-4981-8203-4adb0d1b0340-metrics-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:44:57.995798 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:57.995645 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f0f494d-1bc3-4981-8203-4adb0d1b0340-openshift-service-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:44:57.995798 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:57.995661 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzfsq\" (UniqueName: \"kubernetes.io/projected/8f0f494d-1bc3-4981-8203-4adb0d1b0340-kube-api-access-fzfsq\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:44:58.548018 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:58.547983 4952 generic.go:356] "Generic (PLEG): container finished" podID="8f0f494d-1bc3-4981-8203-4adb0d1b0340" containerID="cd44f411eba882c4bc1f492dd926c35d21629a4cdfebe402105dc7699b8d2c9e" exitCode=0 Oct 30 23:44:58.548506 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:58.548053 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55666687-7659595c45-jwcgn" event={"ID":"8f0f494d-1bc3-4981-8203-4adb0d1b0340","Type":"ContainerDied","Data":"cd44f411eba882c4bc1f492dd926c35d21629a4cdfebe402105dc7699b8d2c9e"} Oct 30 23:44:58.548506 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:58.548069 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55666687-7659595c45-jwcgn" Oct 30 23:44:58.548506 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:58.548121 4952 scope.go:122] "RemoveContainer" containerID="cd44f411eba882c4bc1f492dd926c35d21629a4cdfebe402105dc7699b8d2c9e" Oct 30 23:44:58.548506 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:58.548112 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55666687-7659595c45-jwcgn" event={"ID":"8f0f494d-1bc3-4981-8203-4adb0d1b0340","Type":"ContainerDied","Data":"a2116525a88a3d9d7f5c00971c66aafbcbd9b92561df90a7e4a14e666ec83a11"} Oct 30 23:44:58.560351 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:58.560331 4952 scope.go:122] "RemoveContainer" containerID="cd44f411eba882c4bc1f492dd926c35d21629a4cdfebe402105dc7699b8d2c9e" Oct 30 23:44:58.560664 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:44:58.560641 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd44f411eba882c4bc1f492dd926c35d21629a4cdfebe402105dc7699b8d2c9e\": container with ID starting with cd44f411eba882c4bc1f492dd926c35d21629a4cdfebe402105dc7699b8d2c9e not found: ID does not exist" containerID="cd44f411eba882c4bc1f492dd926c35d21629a4cdfebe402105dc7699b8d2c9e" Oct 30 23:44:58.560761 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:58.560671 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd44f411eba882c4bc1f492dd926c35d21629a4cdfebe402105dc7699b8d2c9e"} err="failed to get container status \"cd44f411eba882c4bc1f492dd926c35d21629a4cdfebe402105dc7699b8d2c9e\": rpc error: code = NotFound desc = could not find container \"cd44f411eba882c4bc1f492dd926c35d21629a4cdfebe402105dc7699b8d2c9e\": container with ID starting with cd44f411eba882c4bc1f492dd926c35d21629a4cdfebe402105dc7699b8d2c9e not found: ID does not exist" Oct 30 23:44:58.719119 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:58.715769 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-55666687-7659595c45-jwcgn"] Oct 30 23:44:58.764079 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:58.764019 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-55666687-7659595c45-jwcgn"] Oct 30 23:44:58.971496 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:58.971415 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f0f494d-1bc3-4981-8203-4adb0d1b0340" path="/var/lib/kubelet/pods/8f0f494d-1bc3-4981-8203-4adb0d1b0340/volumes" Oct 30 23:44:59.296582 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:59.296505 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:59.570377 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:59.570192 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:44:59.801429 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:44:59.801394 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:00.166341 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.166312 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq"] Oct 30 23:45:00.166873 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.166860 4952 cpu_manager.go:414] "containerMap: removing container" podUID="b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4" containerName="payload-pre-processing" Oct 30 23:45:00.166918 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.166875 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4" containerName="payload-pre-processing" Oct 30 23:45:00.166918 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.166900 4952 cpu_manager.go:414] "containerMap: removing container" podUID="86ec338e-ad13-4029-a1cc-88650c6a27ca" containerName="main" Oct 30 23:45:00.166918 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.166905 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86ec338e-ad13-4029-a1cc-88650c6a27ca" containerName="main" Oct 30 23:45:00.166918 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.166913 4952 cpu_manager.go:414] "containerMap: removing container" podUID="bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04" containerName="maas-api" Oct 30 23:45:00.166918 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.166918 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04" containerName="maas-api" Oct 30 23:45:00.167109 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.166926 4952 cpu_manager.go:414] "containerMap: removing container" podUID="8f0f494d-1bc3-4981-8203-4adb0d1b0340" containerName="maas-api" Oct 30 23:45:00.167109 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.166930 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f0f494d-1bc3-4981-8203-4adb0d1b0340" containerName="maas-api" Oct 30 23:45:00.167109 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.166942 4952 cpu_manager.go:414] "containerMap: removing container" podUID="b155a013-f16a-481e-a4f0-a1afd8140a9f" containerName="maas-api" Oct 30 23:45:00.167109 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.166946 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b155a013-f16a-481e-a4f0-a1afd8140a9f" containerName="maas-api" Oct 30 23:45:00.167109 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.166953 4952 cpu_manager.go:414] "containerMap: removing container" podUID="88e27e01-b433-41c6-b19c-657bcb771e1c" containerName="maas-api" Oct 30 23:45:00.167109 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.166958 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88e27e01-b433-41c6-b19c-657bcb771e1c" containerName="maas-api" Oct 30 23:45:00.167109 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.167090 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8636dbc-385a-4b9f-a8ff-f0c40a87a6a4" containerName="payload-pre-processing" Oct 30 23:45:00.167109 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.167103 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="86ec338e-ad13-4029-a1cc-88650c6a27ca" containerName="main" Oct 30 23:45:00.167109 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.167110 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb5d297e-1e95-4e95-b2d5-7c8aad4e7a04" containerName="maas-api" Oct 30 23:45:00.167367 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.167118 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f0f494d-1bc3-4981-8203-4adb0d1b0340" containerName="maas-api" Oct 30 23:45:00.167367 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.167124 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="b155a013-f16a-481e-a4f0-a1afd8140a9f" containerName="maas-api" Oct 30 23:45:00.167367 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.167131 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="88e27e01-b433-41c6-b19c-657bcb771e1c" containerName="maas-api" Oct 30 23:45:00.171915 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.171895 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq" Oct 30 23:45:00.193055 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.193013 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m"] Oct 30 23:45:00.198389 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.198367 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j"] Oct 30 23:45:00.198792 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.198680 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m" Oct 30 23:45:00.203234 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.203215 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j" Oct 30 23:45:00.221638 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.218382 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm"] Oct 30 23:45:00.224890 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.224865 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm" Oct 30 23:45:00.232575 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.232378 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cxnq\" (UniqueName: \"kubernetes.io/projected/54e5ee1a-c524-4f52-bc9c-870411e7f17e-kube-api-access-2cxnq\") pod \"maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq\" (UID: \"54e5ee1a-c524-4f52-bc9c-870411e7f17e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq" Oct 30 23:45:00.232717 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.232609 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e5ee1a-c524-4f52-bc9c-870411e7f17e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq\" (UID: \"54e5ee1a-c524-4f52-bc9c-870411e7f17e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq" Oct 30 23:45:00.236276 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.236243 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:00.239859 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.239840 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\"" Oct 30 23:45:00.278836 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.278800 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq"] Oct 30 23:45:00.280132 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.280103 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m"] Oct 30 23:45:00.285638 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.285612 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j"] Oct 30 23:45:00.336855 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.336796 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwv6l\" (UniqueName: \"kubernetes.io/projected/032662ea-6f38-4092-8bda-d27d09daf88f-kube-api-access-xwv6l\") pod \"maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm\" (UID: \"032662ea-6f38-4092-8bda-d27d09daf88f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm" Oct 30 23:45:00.337068 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.336923 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cxnq\" (UniqueName: \"kubernetes.io/projected/54e5ee1a-c524-4f52-bc9c-870411e7f17e-kube-api-access-2cxnq\") pod \"maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq\" (UID: \"54e5ee1a-c524-4f52-bc9c-870411e7f17e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq" Oct 30 23:45:00.337068 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.336994 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jtqb\" (UniqueName: \"kubernetes.io/projected/89959083-95eb-4626-b1b2-4b7ac9d76840-kube-api-access-8jtqb\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m\" (UID: \"89959083-95eb-4626-b1b2-4b7ac9d76840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m" Oct 30 23:45:00.337227 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.337135 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd22a14-b583-4991-8ced-45c5b15919ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j\" (UID: \"5cd22a14-b583-4991-8ced-45c5b15919ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j" Oct 30 23:45:00.337227 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.337177 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89959083-95eb-4626-b1b2-4b7ac9d76840-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m\" (UID: \"89959083-95eb-4626-b1b2-4b7ac9d76840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m" Oct 30 23:45:00.337332 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.337264 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/032662ea-6f38-4092-8bda-d27d09daf88f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm\" (UID: \"032662ea-6f38-4092-8bda-d27d09daf88f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm" Oct 30 23:45:00.337332 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.337296 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqgcj\" (UniqueName: \"kubernetes.io/projected/5cd22a14-b583-4991-8ced-45c5b15919ba-kube-api-access-sqgcj\") pod \"maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j\" (UID: \"5cd22a14-b583-4991-8ced-45c5b15919ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j" Oct 30 23:45:00.337459 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.337345 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e5ee1a-c524-4f52-bc9c-870411e7f17e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq\" (UID: \"54e5ee1a-c524-4f52-bc9c-870411e7f17e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq" Oct 30 23:45:00.338080 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.338056 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e5ee1a-c524-4f52-bc9c-870411e7f17e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq\" (UID: \"54e5ee1a-c524-4f52-bc9c-870411e7f17e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq" Oct 30 23:45:00.338813 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.338793 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm"] Oct 30 23:45:00.394991 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.394960 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cxnq\" (UniqueName: \"kubernetes.io/projected/54e5ee1a-c524-4f52-bc9c-870411e7f17e-kube-api-access-2cxnq\") pod \"maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq\" (UID: \"54e5ee1a-c524-4f52-bc9c-870411e7f17e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq" Oct 30 23:45:00.438260 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.438206 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/032662ea-6f38-4092-8bda-d27d09daf88f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm\" (UID: \"032662ea-6f38-4092-8bda-d27d09daf88f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm" Oct 30 23:45:00.438260 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.438237 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqgcj\" (UniqueName: \"kubernetes.io/projected/5cd22a14-b583-4991-8ced-45c5b15919ba-kube-api-access-sqgcj\") pod \"maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j\" (UID: \"5cd22a14-b583-4991-8ced-45c5b15919ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j" Oct 30 23:45:00.438396 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.438305 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwv6l\" (UniqueName: \"kubernetes.io/projected/032662ea-6f38-4092-8bda-d27d09daf88f-kube-api-access-xwv6l\") pod \"maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm\" (UID: \"032662ea-6f38-4092-8bda-d27d09daf88f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm" Oct 30 23:45:00.438396 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.438362 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jtqb\" (UniqueName: \"kubernetes.io/projected/89959083-95eb-4626-b1b2-4b7ac9d76840-kube-api-access-8jtqb\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m\" (UID: \"89959083-95eb-4626-b1b2-4b7ac9d76840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m" Oct 30 23:45:00.438495 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.438407 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd22a14-b583-4991-8ced-45c5b15919ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j\" (UID: \"5cd22a14-b583-4991-8ced-45c5b15919ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j" Oct 30 23:45:00.438495 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.438429 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89959083-95eb-4626-b1b2-4b7ac9d76840-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m\" (UID: \"89959083-95eb-4626-b1b2-4b7ac9d76840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m" Oct 30 23:45:00.438889 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.438869 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/032662ea-6f38-4092-8bda-d27d09daf88f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm\" (UID: \"032662ea-6f38-4092-8bda-d27d09daf88f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm" Oct 30 23:45:00.439217 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.439196 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89959083-95eb-4626-b1b2-4b7ac9d76840-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m\" (UID: \"89959083-95eb-4626-b1b2-4b7ac9d76840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m" Oct 30 23:45:00.439265 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.439204 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd22a14-b583-4991-8ced-45c5b15919ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j\" (UID: \"5cd22a14-b583-4991-8ced-45c5b15919ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j" Oct 30 23:45:00.486905 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.486882 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq" Oct 30 23:45:00.496456 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.496428 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwv6l\" (UniqueName: \"kubernetes.io/projected/032662ea-6f38-4092-8bda-d27d09daf88f-kube-api-access-xwv6l\") pod \"maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm\" (UID: \"032662ea-6f38-4092-8bda-d27d09daf88f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm" Oct 30 23:45:00.496594 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.496573 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqgcj\" (UniqueName: \"kubernetes.io/projected/5cd22a14-b583-4991-8ced-45c5b15919ba-kube-api-access-sqgcj\") pod \"maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j\" (UID: \"5cd22a14-b583-4991-8ced-45c5b15919ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j" Oct 30 23:45:00.496655 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.496590 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jtqb\" (UniqueName: \"kubernetes.io/projected/89959083-95eb-4626-b1b2-4b7ac9d76840-kube-api-access-8jtqb\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m\" (UID: \"89959083-95eb-4626-b1b2-4b7ac9d76840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m" Oct 30 23:45:00.516699 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.516677 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m" Oct 30 23:45:00.523398 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.523364 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" podUID="28913940-90f7-4d80-89b7-f5eece975d02" containerName="main" containerID="cri-o://56504e0ec32c81c45b4e864516ca85c7c433d760801756c25a038e67a15606ba" gracePeriod=45 Oct 30 23:45:00.544308 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.544243 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j" Oct 30 23:45:00.556427 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.556279 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm" Oct 30 23:45:00.581111 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.578837 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" podUID="43e39353-ae85-4d76-9900-7e7927409098" containerName="main" containerID="cri-o://70f31c88b1be0dd73610bd06a5d006b036cafec90cad1585d4560c958db9493d" gracePeriod=45 Oct 30 23:45:00.581111 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.578996 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" podUID="44adfd6a-2626-4378-bca1-d46ee1670b07" containerName="main" containerID="cri-o://1654b012d1b730807798a9ab3f575b0c4504c87d2e0b5af0dac47148aaf91c17" gracePeriod=45 Oct 30 23:45:00.592762 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.592733 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" podUID="43e39353-ae85-4d76-9900-7e7927409098" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": dial tcp 10.132.0.70:8000: connect: connection refused" Oct 30 23:45:00.610544 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.608864 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" podUID="44adfd6a-2626-4378-bca1-d46ee1670b07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": dial tcp 10.132.0.73:8000: connect: connection refused" Oct 30 23:45:00.879925 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.879882 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq"] Oct 30 23:45:00.883283 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:45:00.880815 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54e5ee1a_c524_4f52_bc9c_870411e7f17e.slice/crio-acf68f8cc1f03601f055fd8e056d173609e9b85c99ec9abae8bfc5b23c4da3d8 WatchSource:0}: Error finding container acf68f8cc1f03601f055fd8e056d173609e9b85c99ec9abae8bfc5b23c4da3d8: Status 404 returned error can't find the container with id acf68f8cc1f03601f055fd8e056d173609e9b85c99ec9abae8bfc5b23c4da3d8 Oct 30 23:45:00.925477 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.925453 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" Oct 30 23:45:00.939243 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.939216 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" Oct 30 23:45:00.944104 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.944079 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" Oct 30 23:45:00.953501 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:45:00.953476 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89959083_95eb_4626_b1b2_4b7ac9d76840.slice/crio-5149a838a32f312817638c9a60698e92f91b86e1d28b4074f5f03c36d23e9e46 WatchSource:0}: Error finding container 5149a838a32f312817638c9a60698e92f91b86e1d28b4074f5f03c36d23e9e46: Status 404 returned error can't find the container with id 5149a838a32f312817638c9a60698e92f91b86e1d28b4074f5f03c36d23e9e46 Oct 30 23:45:00.955795 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.955776 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m"] Oct 30 23:45:00.988087 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:00.988011 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j"] Oct 30 23:45:01.005173 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.005150 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm"] Oct 30 23:45:01.006465 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:45:01.006442 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod032662ea_6f38_4092_8bda_d27d09daf88f.slice/crio-4857a6b5d8b55f643c365cd5f2c1374cd842567955c192cec8370b3930116db8 WatchSource:0}: Error finding container 4857a6b5d8b55f643c365cd5f2c1374cd842567955c192cec8370b3930116db8: Status 404 returned error can't find the container with id 4857a6b5d8b55f643c365cd5f2c1374cd842567955c192cec8370b3930116db8 Oct 30 23:45:01.052081 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.052005 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43e39353-ae85-4d76-9900-7e7927409098-dshm\" (UniqueName: \"kubernetes.io/empty-dir/43e39353-ae85-4d76-9900-7e7927409098-dshm\") pod \"43e39353-ae85-4d76-9900-7e7927409098\" (UID: \"43e39353-ae85-4d76-9900-7e7927409098\") " Oct 30 23:45:01.052250 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.052083 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28913940-90f7-4d80-89b7-f5eece975d02-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28913940-90f7-4d80-89b7-f5eece975d02-model-cache\") pod \"28913940-90f7-4d80-89b7-f5eece975d02\" (UID: \"28913940-90f7-4d80-89b7-f5eece975d02\") " Oct 30 23:45:01.052250 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.052139 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44adfd6a-2626-4378-bca1-d46ee1670b07-home\" (UniqueName: \"kubernetes.io/empty-dir/44adfd6a-2626-4378-bca1-d46ee1670b07-home\") pod \"44adfd6a-2626-4378-bca1-d46ee1670b07\" (UID: \"44adfd6a-2626-4378-bca1-d46ee1670b07\") " Oct 30 23:45:01.052250 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.052179 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44adfd6a-2626-4378-bca1-d46ee1670b07-tls-certs\" (UniqueName: \"kubernetes.io/secret/44adfd6a-2626-4378-bca1-d46ee1670b07-tls-certs\") pod \"44adfd6a-2626-4378-bca1-d46ee1670b07\" (UID: \"44adfd6a-2626-4378-bca1-d46ee1670b07\") " Oct 30 23:45:01.052250 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.052220 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28913940-90f7-4d80-89b7-f5eece975d02-tls-certs\" (UniqueName: \"kubernetes.io/secret/28913940-90f7-4d80-89b7-f5eece975d02-tls-certs\") pod \"28913940-90f7-4d80-89b7-f5eece975d02\" (UID: \"28913940-90f7-4d80-89b7-f5eece975d02\") " Oct 30 23:45:01.052486 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.052258 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28913940-90f7-4d80-89b7-f5eece975d02-dshm\" (UniqueName: \"kubernetes.io/empty-dir/28913940-90f7-4d80-89b7-f5eece975d02-dshm\") pod \"28913940-90f7-4d80-89b7-f5eece975d02\" (UID: \"28913940-90f7-4d80-89b7-f5eece975d02\") " Oct 30 23:45:01.052486 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.052320 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43e39353-ae85-4d76-9900-7e7927409098-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43e39353-ae85-4d76-9900-7e7927409098-tmp-dir\") pod \"43e39353-ae85-4d76-9900-7e7927409098\" (UID: \"43e39353-ae85-4d76-9900-7e7927409098\") " Oct 30 23:45:01.052486 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.052369 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43e39353-ae85-4d76-9900-7e7927409098-kube-api-access-s5wxm\" (UniqueName: \"kubernetes.io/projected/43e39353-ae85-4d76-9900-7e7927409098-kube-api-access-s5wxm\") pod \"43e39353-ae85-4d76-9900-7e7927409098\" (UID: \"43e39353-ae85-4d76-9900-7e7927409098\") " Oct 30 23:45:01.052486 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.052402 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28913940-90f7-4d80-89b7-f5eece975d02-home\" (UniqueName: \"kubernetes.io/empty-dir/28913940-90f7-4d80-89b7-f5eece975d02-home\") pod \"28913940-90f7-4d80-89b7-f5eece975d02\" (UID: \"28913940-90f7-4d80-89b7-f5eece975d02\") " Oct 30 23:45:01.052486 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.052435 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28913940-90f7-4d80-89b7-f5eece975d02-kube-api-access-px427\" (UniqueName: \"kubernetes.io/projected/28913940-90f7-4d80-89b7-f5eece975d02-kube-api-access-px427\") pod \"28913940-90f7-4d80-89b7-f5eece975d02\" (UID: \"28913940-90f7-4d80-89b7-f5eece975d02\") " Oct 30 23:45:01.052486 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.052472 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43e39353-ae85-4d76-9900-7e7927409098-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43e39353-ae85-4d76-9900-7e7927409098-model-cache\") pod \"43e39353-ae85-4d76-9900-7e7927409098\" (UID: \"43e39353-ae85-4d76-9900-7e7927409098\") " Oct 30 23:45:01.052811 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.052538 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44adfd6a-2626-4378-bca1-d46ee1670b07-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44adfd6a-2626-4378-bca1-d46ee1670b07-model-cache\") pod \"44adfd6a-2626-4378-bca1-d46ee1670b07\" (UID: \"44adfd6a-2626-4378-bca1-d46ee1670b07\") " Oct 30 23:45:01.052811 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.052578 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44adfd6a-2626-4378-bca1-d46ee1670b07-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44adfd6a-2626-4378-bca1-d46ee1670b07-tmp-dir\") pod \"44adfd6a-2626-4378-bca1-d46ee1670b07\" (UID: \"44adfd6a-2626-4378-bca1-d46ee1670b07\") " Oct 30 23:45:01.052960 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.052859 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44adfd6a-2626-4378-bca1-d46ee1670b07-tmp-dir" pod "44adfd6a-2626-4378-bca1-d46ee1670b07" (UID: "44adfd6a-2626-4378-bca1-d46ee1670b07"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:45:01.053109 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.052899 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44adfd6a-2626-4378-bca1-d46ee1670b07-dshm\" (UniqueName: \"kubernetes.io/empty-dir/44adfd6a-2626-4378-bca1-d46ee1670b07-dshm\") pod \"44adfd6a-2626-4378-bca1-d46ee1670b07\" (UID: \"44adfd6a-2626-4378-bca1-d46ee1670b07\") " Oct 30 23:45:01.053197 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.053179 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43e39353-ae85-4d76-9900-7e7927409098-home\" (UniqueName: \"kubernetes.io/empty-dir/43e39353-ae85-4d76-9900-7e7927409098-home\") pod \"43e39353-ae85-4d76-9900-7e7927409098\" (UID: \"43e39353-ae85-4d76-9900-7e7927409098\") " Oct 30 23:45:01.053268 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.053227 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28913940-90f7-4d80-89b7-f5eece975d02-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28913940-90f7-4d80-89b7-f5eece975d02-tmp-dir\") pod \"28913940-90f7-4d80-89b7-f5eece975d02\" (UID: \"28913940-90f7-4d80-89b7-f5eece975d02\") " Oct 30 23:45:01.053331 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.053268 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44adfd6a-2626-4378-bca1-d46ee1670b07-kube-api-access-dlfks\" (UniqueName: \"kubernetes.io/projected/44adfd6a-2626-4378-bca1-d46ee1670b07-kube-api-access-dlfks\") pod \"44adfd6a-2626-4378-bca1-d46ee1670b07\" (UID: \"44adfd6a-2626-4378-bca1-d46ee1670b07\") " Oct 30 23:45:01.053395 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.053347 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43e39353-ae85-4d76-9900-7e7927409098-tls-certs\" (UniqueName: \"kubernetes.io/secret/43e39353-ae85-4d76-9900-7e7927409098-tls-certs\") pod \"43e39353-ae85-4d76-9900-7e7927409098\" (UID: \"43e39353-ae85-4d76-9900-7e7927409098\") " Oct 30 23:45:01.053968 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.053945 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44adfd6a-2626-4378-bca1-d46ee1670b07-tmp-dir\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:45:01.054493 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.054467 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e39353-ae85-4d76-9900-7e7927409098-tmp-dir" pod "43e39353-ae85-4d76-9900-7e7927409098" (UID: "43e39353-ae85-4d76-9900-7e7927409098"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:45:01.054821 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.054665 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28913940-90f7-4d80-89b7-f5eece975d02-model-cache" pod "28913940-90f7-4d80-89b7-f5eece975d02" (UID: "28913940-90f7-4d80-89b7-f5eece975d02"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:45:01.054990 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.054964 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44adfd6a-2626-4378-bca1-d46ee1670b07-home" pod "44adfd6a-2626-4378-bca1-d46ee1670b07" (UID: "44adfd6a-2626-4378-bca1-d46ee1670b07"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:45:01.056114 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.056047 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e39353-ae85-4d76-9900-7e7927409098-dshm" pod "43e39353-ae85-4d76-9900-7e7927409098" (UID: "43e39353-ae85-4d76-9900-7e7927409098"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:45:01.056612 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.056450 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e39353-ae85-4d76-9900-7e7927409098-model-cache" pod "43e39353-ae85-4d76-9900-7e7927409098" (UID: "43e39353-ae85-4d76-9900-7e7927409098"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:45:01.056898 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.056746 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44adfd6a-2626-4378-bca1-d46ee1670b07-model-cache" pod "44adfd6a-2626-4378-bca1-d46ee1670b07" (UID: "44adfd6a-2626-4378-bca1-d46ee1670b07"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:45:01.057539 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.057513 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e39353-ae85-4d76-9900-7e7927409098-home" pod "43e39353-ae85-4d76-9900-7e7927409098" (UID: "43e39353-ae85-4d76-9900-7e7927409098"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:45:01.057816 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.057793 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28913940-90f7-4d80-89b7-f5eece975d02-home" pod "28913940-90f7-4d80-89b7-f5eece975d02" (UID: "28913940-90f7-4d80-89b7-f5eece975d02"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:45:01.057910 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.057879 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28913940-90f7-4d80-89b7-f5eece975d02-tmp-dir" pod "28913940-90f7-4d80-89b7-f5eece975d02" (UID: "28913940-90f7-4d80-89b7-f5eece975d02"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:45:01.058091 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.058067 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44adfd6a-2626-4378-bca1-d46ee1670b07-dshm" pod "44adfd6a-2626-4378-bca1-d46ee1670b07" (UID: "44adfd6a-2626-4378-bca1-d46ee1670b07"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:45:01.058449 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.058424 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44adfd6a-2626-4378-bca1-d46ee1670b07-tls-certs" pod "44adfd6a-2626-4378-bca1-d46ee1670b07" (UID: "44adfd6a-2626-4378-bca1-d46ee1670b07"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:45:01.058548 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.058533 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28913940-90f7-4d80-89b7-f5eece975d02-kube-api-access-px427" pod "28913940-90f7-4d80-89b7-f5eece975d02" (UID: "28913940-90f7-4d80-89b7-f5eece975d02"). InnerVolumeSpecName "kube-api-access-px427". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:45:01.059105 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.059014 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43e39353-ae85-4d76-9900-7e7927409098-tls-certs" pod "43e39353-ae85-4d76-9900-7e7927409098" (UID: "43e39353-ae85-4d76-9900-7e7927409098"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:45:01.059676 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.059642 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28913940-90f7-4d80-89b7-f5eece975d02-tls-certs" pod "28913940-90f7-4d80-89b7-f5eece975d02" (UID: "28913940-90f7-4d80-89b7-f5eece975d02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:45:01.060480 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.060454 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44adfd6a-2626-4378-bca1-d46ee1670b07-kube-api-access-dlfks" pod "44adfd6a-2626-4378-bca1-d46ee1670b07" (UID: "44adfd6a-2626-4378-bca1-d46ee1670b07"). InnerVolumeSpecName "kube-api-access-dlfks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:45:01.061384 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.061356 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28913940-90f7-4d80-89b7-f5eece975d02-dshm" pod "28913940-90f7-4d80-89b7-f5eece975d02" (UID: "28913940-90f7-4d80-89b7-f5eece975d02"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:45:01.061470 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.061387 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e39353-ae85-4d76-9900-7e7927409098-kube-api-access-s5wxm" pod "43e39353-ae85-4d76-9900-7e7927409098" (UID: "43e39353-ae85-4d76-9900-7e7927409098"). InnerVolumeSpecName "kube-api-access-s5wxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:45:01.155366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.155107 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43e39353-ae85-4d76-9900-7e7927409098-home\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:45:01.155366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.155145 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28913940-90f7-4d80-89b7-f5eece975d02-tmp-dir\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:45:01.155366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.155158 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlfks\" (UniqueName: \"kubernetes.io/projected/44adfd6a-2626-4378-bca1-d46ee1670b07-kube-api-access-dlfks\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:45:01.155366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.155173 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43e39353-ae85-4d76-9900-7e7927409098-tls-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:45:01.155366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.155185 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43e39353-ae85-4d76-9900-7e7927409098-dshm\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:45:01.155366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.155197 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28913940-90f7-4d80-89b7-f5eece975d02-model-cache\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:45:01.155366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.155209 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44adfd6a-2626-4378-bca1-d46ee1670b07-home\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:45:01.155366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.155222 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44adfd6a-2626-4378-bca1-d46ee1670b07-tls-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:45:01.155366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.155234 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28913940-90f7-4d80-89b7-f5eece975d02-tls-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:45:01.155366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.155246 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28913940-90f7-4d80-89b7-f5eece975d02-dshm\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:45:01.155366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.155259 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43e39353-ae85-4d76-9900-7e7927409098-tmp-dir\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:45:01.155366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.155270 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5wxm\" (UniqueName: \"kubernetes.io/projected/43e39353-ae85-4d76-9900-7e7927409098-kube-api-access-s5wxm\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:45:01.155366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.155282 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28913940-90f7-4d80-89b7-f5eece975d02-home\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:45:01.155366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.155297 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px427\" (UniqueName: \"kubernetes.io/projected/28913940-90f7-4d80-89b7-f5eece975d02-kube-api-access-px427\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:45:01.155366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.155308 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43e39353-ae85-4d76-9900-7e7927409098-model-cache\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:45:01.155366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.155323 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44adfd6a-2626-4378-bca1-d46ee1670b07-model-cache\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:45:01.155366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.155335 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44adfd6a-2626-4378-bca1-d46ee1670b07-dshm\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:45:01.592409 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.592376 4952 generic.go:356] "Generic (PLEG): container finished" podID="43e39353-ae85-4d76-9900-7e7927409098" containerID="70f31c88b1be0dd73610bd06a5d006b036cafec90cad1585d4560c958db9493d" exitCode=0 Oct 30 23:45:01.592882 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.592428 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" Oct 30 23:45:01.592882 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.592456 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" event={"ID":"43e39353-ae85-4d76-9900-7e7927409098","Type":"ContainerDied","Data":"70f31c88b1be0dd73610bd06a5d006b036cafec90cad1585d4560c958db9493d"} Oct 30 23:45:01.592882 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.592501 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll" event={"ID":"43e39353-ae85-4d76-9900-7e7927409098","Type":"ContainerDied","Data":"dbab0483aeef4c494d22f349bd64d1f61766db3c24bf2dbd64c16e03748c8155"} Oct 30 23:45:01.592882 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.592526 4952 scope.go:122] "RemoveContainer" containerID="70f31c88b1be0dd73610bd06a5d006b036cafec90cad1585d4560c958db9493d" Oct 30 23:45:01.594525 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.594500 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm" event={"ID":"032662ea-6f38-4092-8bda-d27d09daf88f","Type":"ContainerStarted","Data":"2e76a5613d7ff1058035ee076e39792322204a83d86b97f627d42dbde6b322b6"} Oct 30 23:45:01.594640 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.594549 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm" event={"ID":"032662ea-6f38-4092-8bda-d27d09daf88f","Type":"ContainerStarted","Data":"4857a6b5d8b55f643c365cd5f2c1374cd842567955c192cec8370b3930116db8"} Oct 30 23:45:01.596668 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.596643 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq" event={"ID":"54e5ee1a-c524-4f52-bc9c-870411e7f17e","Type":"ContainerStarted","Data":"5dd83ea01edcb7661519c7f8afec6e62b2a140ab4b5b310ba924b93eb22b6714"} Oct 30 23:45:01.596798 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.596705 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq" event={"ID":"54e5ee1a-c524-4f52-bc9c-870411e7f17e","Type":"ContainerStarted","Data":"acf68f8cc1f03601f055fd8e056d173609e9b85c99ec9abae8bfc5b23c4da3d8"} Oct 30 23:45:01.598437 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.598411 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m" event={"ID":"89959083-95eb-4626-b1b2-4b7ac9d76840","Type":"ContainerStarted","Data":"9dc21a962afce62daa5b6f5c320e0f4d81dac9d0a7dea6ee9ed5d19191b3f5f6"} Oct 30 23:45:01.598528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.598470 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m" event={"ID":"89959083-95eb-4626-b1b2-4b7ac9d76840","Type":"ContainerStarted","Data":"5149a838a32f312817638c9a60698e92f91b86e1d28b4074f5f03c36d23e9e46"} Oct 30 23:45:01.600120 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.600094 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j" event={"ID":"5cd22a14-b583-4991-8ced-45c5b15919ba","Type":"ContainerStarted","Data":"35385385c552a1cb7e7dfb711a6573196bc8ccacdb398571f2dbfa71ccab80f8"} Oct 30 23:45:01.600223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.600143 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j" event={"ID":"5cd22a14-b583-4991-8ced-45c5b15919ba","Type":"ContainerStarted","Data":"5377ecefe5f6e33b61957d3f22b33cc77f4e440bb42bc02399ef87bc72ffe464"} Oct 30 23:45:01.601856 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.601741 4952 generic.go:356] "Generic (PLEG): container finished" podID="28913940-90f7-4d80-89b7-f5eece975d02" containerID="56504e0ec32c81c45b4e864516ca85c7c433d760801756c25a038e67a15606ba" exitCode=0 Oct 30 23:45:01.601856 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.601774 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" Oct 30 23:45:01.601991 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.601852 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" event={"ID":"28913940-90f7-4d80-89b7-f5eece975d02","Type":"ContainerDied","Data":"56504e0ec32c81c45b4e864516ca85c7c433d760801756c25a038e67a15606ba"} Oct 30 23:45:01.601991 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.601887 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" event={"ID":"28913940-90f7-4d80-89b7-f5eece975d02","Type":"ContainerDied","Data":"3e66f279c6ffdb315bd26749946400c4cf4a49a1d6e75370c8be757c6d656c89"} Oct 30 23:45:01.603516 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.603493 4952 generic.go:356] "Generic (PLEG): container finished" podID="44adfd6a-2626-4378-bca1-d46ee1670b07" containerID="1654b012d1b730807798a9ab3f575b0c4504c87d2e0b5af0dac47148aaf91c17" exitCode=0 Oct 30 23:45:01.603682 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.603558 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" Oct 30 23:45:01.603793 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.603570 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" event={"ID":"44adfd6a-2626-4378-bca1-d46ee1670b07","Type":"ContainerDied","Data":"1654b012d1b730807798a9ab3f575b0c4504c87d2e0b5af0dac47148aaf91c17"} Oct 30 23:45:01.603837 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.603794 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg" event={"ID":"44adfd6a-2626-4378-bca1-d46ee1670b07","Type":"ContainerDied","Data":"4419089e86a9934100686a503163f262d716d104cb84d92fe8db20f1b4883a9c"} Oct 30 23:45:01.605816 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.605793 4952 scope.go:122] "RemoveContainer" containerID="70f31c88b1be0dd73610bd06a5d006b036cafec90cad1585d4560c958db9493d" Oct 30 23:45:01.606144 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:45:01.606117 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70f31c88b1be0dd73610bd06a5d006b036cafec90cad1585d4560c958db9493d\": container with ID starting with 70f31c88b1be0dd73610bd06a5d006b036cafec90cad1585d4560c958db9493d not found: ID does not exist" containerID="70f31c88b1be0dd73610bd06a5d006b036cafec90cad1585d4560c958db9493d" Oct 30 23:45:01.606212 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.606157 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70f31c88b1be0dd73610bd06a5d006b036cafec90cad1585d4560c958db9493d"} err="failed to get container status \"70f31c88b1be0dd73610bd06a5d006b036cafec90cad1585d4560c958db9493d\": rpc error: code = NotFound desc = could not find container \"70f31c88b1be0dd73610bd06a5d006b036cafec90cad1585d4560c958db9493d\": container with ID starting with 70f31c88b1be0dd73610bd06a5d006b036cafec90cad1585d4560c958db9493d not found: ID does not exist" Oct 30 23:45:01.606212 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.606180 4952 scope.go:122] "RemoveContainer" containerID="56504e0ec32c81c45b4e864516ca85c7c433d760801756c25a038e67a15606ba" Oct 30 23:45:01.616950 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.616935 4952 scope.go:122] "RemoveContainer" containerID="56504e0ec32c81c45b4e864516ca85c7c433d760801756c25a038e67a15606ba" Oct 30 23:45:01.617227 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:45:01.617209 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56504e0ec32c81c45b4e864516ca85c7c433d760801756c25a038e67a15606ba\": container with ID starting with 56504e0ec32c81c45b4e864516ca85c7c433d760801756c25a038e67a15606ba not found: ID does not exist" containerID="56504e0ec32c81c45b4e864516ca85c7c433d760801756c25a038e67a15606ba" Oct 30 23:45:01.617292 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.617234 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56504e0ec32c81c45b4e864516ca85c7c433d760801756c25a038e67a15606ba"} err="failed to get container status \"56504e0ec32c81c45b4e864516ca85c7c433d760801756c25a038e67a15606ba\": rpc error: code = NotFound desc = could not find container \"56504e0ec32c81c45b4e864516ca85c7c433d760801756c25a038e67a15606ba\": container with ID starting with 56504e0ec32c81c45b4e864516ca85c7c433d760801756c25a038e67a15606ba not found: ID does not exist" Oct 30 23:45:01.617292 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.617251 4952 scope.go:122] "RemoveContainer" containerID="1654b012d1b730807798a9ab3f575b0c4504c87d2e0b5af0dac47148aaf91c17" Oct 30 23:45:01.626693 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.626664 4952 scope.go:122] "RemoveContainer" containerID="1654b012d1b730807798a9ab3f575b0c4504c87d2e0b5af0dac47148aaf91c17" Oct 30 23:45:01.626956 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:45:01.626938 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1654b012d1b730807798a9ab3f575b0c4504c87d2e0b5af0dac47148aaf91c17\": container with ID starting with 1654b012d1b730807798a9ab3f575b0c4504c87d2e0b5af0dac47148aaf91c17 not found: ID does not exist" containerID="1654b012d1b730807798a9ab3f575b0c4504c87d2e0b5af0dac47148aaf91c17" Oct 30 23:45:01.627026 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.626961 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1654b012d1b730807798a9ab3f575b0c4504c87d2e0b5af0dac47148aaf91c17"} err="failed to get container status \"1654b012d1b730807798a9ab3f575b0c4504c87d2e0b5af0dac47148aaf91c17\": rpc error: code = NotFound desc = could not find container \"1654b012d1b730807798a9ab3f575b0c4504c87d2e0b5af0dac47148aaf91c17\": container with ID starting with 1654b012d1b730807798a9ab3f575b0c4504c87d2e0b5af0dac47148aaf91c17 not found: ID does not exist" Oct 30 23:45:01.776341 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.776287 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm" podStartSLOduration=1.776272991 podStartE2EDuration="1.776272991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:01.774824162 +0000 UTC m=+2320.918654367" watchObservedRunningTime="2026-10-30 23:45:01.776272991 +0000 UTC m=+2320.920103195" Oct 30 23:45:01.799134 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.799094 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz" podUID="28913940-90f7-4d80-89b7-f5eece975d02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:45:01.887295 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.887267 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll"] Oct 30 23:45:01.898056 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:01.893133 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-989a4f/simulator-w4-989a4f-kserve-6f68577976-kkpll"] Oct 30 23:45:02.002318 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:02.002265 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j" podStartSLOduration=2.002249506 podStartE2EDuration="2.002249506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:02.0002513 +0000 UTC m=+2321.144081531" watchObservedRunningTime="2026-10-30 23:45:02.002249506 +0000 UTC m=+2321.146079712" Oct 30 23:45:02.115211 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:02.115165 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m" podStartSLOduration=2.115150615 podStartE2EDuration="2.115150615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:02.111031844 +0000 UTC m=+2321.254862074" watchObservedRunningTime="2026-10-30 23:45:02.115150615 +0000 UTC m=+2321.258980820" Oct 30 23:45:02.223852 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:02.223823 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg"] Oct 30 23:45:02.227237 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:02.227212 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-989a4f/premium-w4-989a4f-kserve-5bdb85b789-s4xcg"] Oct 30 23:45:02.338383 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:02.338334 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq" podStartSLOduration=2.338320625 podStartE2EDuration="2.338320625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:02.334753294 +0000 UTC m=+2321.478583518" watchObservedRunningTime="2026-10-30 23:45:02.338320625 +0000 UTC m=+2321.482150834" Oct 30 23:45:02.446565 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:02.446485 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz"] Oct 30 23:45:02.452093 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:02.452064 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-989a4f/distinct-w4-989a4f-kserve-6bd8f8bdcb-qhrqz"] Oct 30 23:45:02.612167 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:02.612135 4952 generic.go:356] "Generic (PLEG): container finished" podID="5cd22a14-b583-4991-8ced-45c5b15919ba" containerID="35385385c552a1cb7e7dfb711a6573196bc8ccacdb398571f2dbfa71ccab80f8" exitCode=60 Oct 30 23:45:02.612596 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:02.612229 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j" event={"ID":"5cd22a14-b583-4991-8ced-45c5b15919ba","Type":"ContainerDied","Data":"35385385c552a1cb7e7dfb711a6573196bc8ccacdb398571f2dbfa71ccab80f8"} Oct 30 23:45:02.612596 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:02.612474 4952 scope.go:122] "RemoveContainer" containerID="35385385c552a1cb7e7dfb711a6573196bc8ccacdb398571f2dbfa71ccab80f8" Oct 30 23:45:02.617261 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:02.617235 4952 generic.go:356] "Generic (PLEG): container finished" podID="032662ea-6f38-4092-8bda-d27d09daf88f" containerID="2e76a5613d7ff1058035ee076e39792322204a83d86b97f627d42dbde6b322b6" exitCode=60 Oct 30 23:45:02.617393 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:02.617313 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm" event={"ID":"032662ea-6f38-4092-8bda-d27d09daf88f","Type":"ContainerDied","Data":"2e76a5613d7ff1058035ee076e39792322204a83d86b97f627d42dbde6b322b6"} Oct 30 23:45:02.617522 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:02.617508 4952 scope.go:122] "RemoveContainer" containerID="2e76a5613d7ff1058035ee076e39792322204a83d86b97f627d42dbde6b322b6" Oct 30 23:45:02.618963 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:02.618941 4952 generic.go:356] "Generic (PLEG): container finished" podID="54e5ee1a-c524-4f52-bc9c-870411e7f17e" containerID="5dd83ea01edcb7661519c7f8afec6e62b2a140ab4b5b310ba924b93eb22b6714" exitCode=60 Oct 30 23:45:02.619072 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:02.618980 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq" event={"ID":"54e5ee1a-c524-4f52-bc9c-870411e7f17e","Type":"ContainerDied","Data":"5dd83ea01edcb7661519c7f8afec6e62b2a140ab4b5b310ba924b93eb22b6714"} Oct 30 23:45:02.619258 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:02.619243 4952 scope.go:122] "RemoveContainer" containerID="5dd83ea01edcb7661519c7f8afec6e62b2a140ab4b5b310ba924b93eb22b6714" Oct 30 23:45:02.620666 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:02.620648 4952 generic.go:356] "Generic (PLEG): container finished" podID="89959083-95eb-4626-b1b2-4b7ac9d76840" containerID="9dc21a962afce62daa5b6f5c320e0f4d81dac9d0a7dea6ee9ed5d19191b3f5f6" exitCode=60 Oct 30 23:45:02.620777 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:02.620679 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m" event={"ID":"89959083-95eb-4626-b1b2-4b7ac9d76840","Type":"ContainerDied","Data":"9dc21a962afce62daa5b6f5c320e0f4d81dac9d0a7dea6ee9ed5d19191b3f5f6"} Oct 30 23:45:02.620923 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:02.620907 4952 scope.go:122] "RemoveContainer" containerID="9dc21a962afce62daa5b6f5c320e0f4d81dac9d0a7dea6ee9ed5d19191b3f5f6" Oct 30 23:45:02.973343 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:02.973310 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28913940-90f7-4d80-89b7-f5eece975d02" path="/var/lib/kubelet/pods/28913940-90f7-4d80-89b7-f5eece975d02/volumes" Oct 30 23:45:02.973776 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:02.973762 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43e39353-ae85-4d76-9900-7e7927409098" path="/var/lib/kubelet/pods/43e39353-ae85-4d76-9900-7e7927409098/volumes" Oct 30 23:45:02.974204 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:02.974189 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44adfd6a-2626-4378-bca1-d46ee1670b07" path="/var/lib/kubelet/pods/44adfd6a-2626-4378-bca1-d46ee1670b07/volumes" Oct 30 23:45:03.631857 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:03.631826 4952 generic.go:356] "Generic (PLEG): container finished" podID="032662ea-6f38-4092-8bda-d27d09daf88f" containerID="70e36368b60f660664549cfc9281a7ae48a64e458cd8ededd6c93e4a041c5d01" exitCode=60 Oct 30 23:45:03.632319 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:03.631906 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm" event={"ID":"032662ea-6f38-4092-8bda-d27d09daf88f","Type":"ContainerDied","Data":"70e36368b60f660664549cfc9281a7ae48a64e458cd8ededd6c93e4a041c5d01"} Oct 30 23:45:03.632319 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:03.631962 4952 scope.go:122] "RemoveContainer" containerID="2e76a5613d7ff1058035ee076e39792322204a83d86b97f627d42dbde6b322b6" Oct 30 23:45:03.632319 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:03.632258 4952 scope.go:122] "RemoveContainer" containerID="70e36368b60f660664549cfc9281a7ae48a64e458cd8ededd6c93e4a041c5d01" Oct 30 23:45:03.632605 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:45:03.632585 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm_odh-ai-gateway-infra(032662ea-6f38-4092-8bda-d27d09daf88f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm" podUID="032662ea-6f38-4092-8bda-d27d09daf88f" Oct 30 23:45:03.634175 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:03.634151 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq" event={"ID":"54e5ee1a-c524-4f52-bc9c-870411e7f17e","Type":"ContainerStarted","Data":"473f2cb4c743b2d9e4a416aea0321d98f2d6c53610c59bbd123052c110ea2381"} Oct 30 23:45:03.636124 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:03.636097 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m" event={"ID":"89959083-95eb-4626-b1b2-4b7ac9d76840","Type":"ContainerStarted","Data":"d1f240d2a77e59d5b558672b2fc3ff2d08ca64e8c7acd8c8429e229480065069"} Oct 30 23:45:03.638500 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:03.638471 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j" event={"ID":"5cd22a14-b583-4991-8ced-45c5b15919ba","Type":"ContainerStarted","Data":"4ff703cdb460a2b1bfa8a9b4b4bf3aed0025f53d385d76063b2737f41da7ea3b"} Oct 30 23:45:04.647668 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:04.647639 4952 scope.go:122] "RemoveContainer" containerID="70e36368b60f660664549cfc9281a7ae48a64e458cd8ededd6c93e4a041c5d01" Oct 30 23:45:04.648144 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:45:04.647965 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm_odh-ai-gateway-infra(032662ea-6f38-4092-8bda-d27d09daf88f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm" podUID="032662ea-6f38-4092-8bda-d27d09daf88f" Oct 30 23:45:04.649389 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:04.649364 4952 generic.go:356] "Generic (PLEG): container finished" podID="54e5ee1a-c524-4f52-bc9c-870411e7f17e" containerID="473f2cb4c743b2d9e4a416aea0321d98f2d6c53610c59bbd123052c110ea2381" exitCode=60 Oct 30 23:45:04.649503 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:04.649442 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq" event={"ID":"54e5ee1a-c524-4f52-bc9c-870411e7f17e","Type":"ContainerDied","Data":"473f2cb4c743b2d9e4a416aea0321d98f2d6c53610c59bbd123052c110ea2381"} Oct 30 23:45:04.649586 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:04.649539 4952 scope.go:122] "RemoveContainer" containerID="5dd83ea01edcb7661519c7f8afec6e62b2a140ab4b5b310ba924b93eb22b6714" Oct 30 23:45:04.649684 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:04.649666 4952 scope.go:122] "RemoveContainer" containerID="473f2cb4c743b2d9e4a416aea0321d98f2d6c53610c59bbd123052c110ea2381" Oct 30 23:45:04.649983 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:45:04.649956 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq_odh-ai-gateway-infra(54e5ee1a-c524-4f52-bc9c-870411e7f17e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq" podUID="54e5ee1a-c524-4f52-bc9c-870411e7f17e" Oct 30 23:45:04.651630 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:04.651607 4952 generic.go:356] "Generic (PLEG): container finished" podID="89959083-95eb-4626-b1b2-4b7ac9d76840" containerID="d1f240d2a77e59d5b558672b2fc3ff2d08ca64e8c7acd8c8429e229480065069" exitCode=60 Oct 30 23:45:04.651730 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:04.651644 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m" event={"ID":"89959083-95eb-4626-b1b2-4b7ac9d76840","Type":"ContainerDied","Data":"d1f240d2a77e59d5b558672b2fc3ff2d08ca64e8c7acd8c8429e229480065069"} Oct 30 23:45:04.651910 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:04.651898 4952 scope.go:122] "RemoveContainer" containerID="d1f240d2a77e59d5b558672b2fc3ff2d08ca64e8c7acd8c8429e229480065069" Oct 30 23:45:04.652187 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:45:04.652164 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m_odh-ai-gateway-infra(89959083-95eb-4626-b1b2-4b7ac9d76840)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m" podUID="89959083-95eb-4626-b1b2-4b7ac9d76840" Oct 30 23:45:04.653635 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:04.653616 4952 generic.go:356] "Generic (PLEG): container finished" podID="5cd22a14-b583-4991-8ced-45c5b15919ba" containerID="4ff703cdb460a2b1bfa8a9b4b4bf3aed0025f53d385d76063b2737f41da7ea3b" exitCode=60 Oct 30 23:45:04.653733 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:04.653701 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j" event={"ID":"5cd22a14-b583-4991-8ced-45c5b15919ba","Type":"ContainerDied","Data":"4ff703cdb460a2b1bfa8a9b4b4bf3aed0025f53d385d76063b2737f41da7ea3b"} Oct 30 23:45:04.653905 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:04.653892 4952 scope.go:122] "RemoveContainer" containerID="4ff703cdb460a2b1bfa8a9b4b4bf3aed0025f53d385d76063b2737f41da7ea3b" Oct 30 23:45:04.654131 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:45:04.654115 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j_odh-ai-gateway-infra(5cd22a14-b583-4991-8ced-45c5b15919ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j" podUID="5cd22a14-b583-4991-8ced-45c5b15919ba" Oct 30 23:45:04.666111 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:04.664146 4952 scope.go:122] "RemoveContainer" containerID="9dc21a962afce62daa5b6f5c320e0f4d81dac9d0a7dea6ee9ed5d19191b3f5f6" Oct 30 23:45:04.689390 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:04.689349 4952 scope.go:122] "RemoveContainer" containerID="35385385c552a1cb7e7dfb711a6573196bc8ccacdb398571f2dbfa71ccab80f8" Oct 30 23:45:04.711904 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:04.711874 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:04.991865 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:04.991783 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:05.217670 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:05.217593 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:05.663599 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:05.663566 4952 scope.go:122] "RemoveContainer" containerID="d1f240d2a77e59d5b558672b2fc3ff2d08ca64e8c7acd8c8429e229480065069" Oct 30 23:45:05.664085 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:45:05.663880 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m_odh-ai-gateway-infra(89959083-95eb-4626-b1b2-4b7ac9d76840)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m" podUID="89959083-95eb-4626-b1b2-4b7ac9d76840" Oct 30 23:45:05.665289 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:05.665266 4952 scope.go:122] "RemoveContainer" containerID="4ff703cdb460a2b1bfa8a9b4b4bf3aed0025f53d385d76063b2737f41da7ea3b" Oct 30 23:45:05.665546 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:45:05.665526 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j_odh-ai-gateway-infra(5cd22a14-b583-4991-8ced-45c5b15919ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j" podUID="5cd22a14-b583-4991-8ced-45c5b15919ba" Oct 30 23:45:05.666979 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:05.666962 4952 scope.go:122] "RemoveContainer" containerID="473f2cb4c743b2d9e4a416aea0321d98f2d6c53610c59bbd123052c110ea2381" Oct 30 23:45:05.667194 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:45:05.667176 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq_odh-ai-gateway-infra(54e5ee1a-c524-4f52-bc9c-870411e7f17e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq" podUID="54e5ee1a-c524-4f52-bc9c-870411e7f17e" Oct 30 23:45:08.947612 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:08.947574 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:09.209490 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:09.209390 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:09.993181 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:09.992064 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:10.211429 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:10.211390 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:14.851652 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:14.851604 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:14.964116 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:14.964091 4952 scope.go:122] "RemoveContainer" containerID="473f2cb4c743b2d9e4a416aea0321d98f2d6c53610c59bbd123052c110ea2381" Oct 30 23:45:15.741305 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:15.741271 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq" event={"ID":"54e5ee1a-c524-4f52-bc9c-870411e7f17e","Type":"ContainerStarted","Data":"4506846db209ea49c809183ab55de82d43986fe129c4f71bde61019dae70ed76"} Oct 30 23:45:15.947748 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:15.947720 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:15.964067 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:15.964017 4952 scope.go:122] "RemoveContainer" containerID="70e36368b60f660664549cfc9281a7ae48a64e458cd8ededd6c93e4a041c5d01" Oct 30 23:45:16.174707 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:16.174675 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:16.750590 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:16.750548 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm" event={"ID":"032662ea-6f38-4092-8bda-d27d09daf88f","Type":"ContainerStarted","Data":"53e98f885e9bfa17cdd347b2275190fdcac91350d16cb2cd5a27ee2ff128a656"} Oct 30 23:45:16.752490 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:16.752465 4952 generic.go:356] "Generic (PLEG): container finished" podID="54e5ee1a-c524-4f52-bc9c-870411e7f17e" containerID="4506846db209ea49c809183ab55de82d43986fe129c4f71bde61019dae70ed76" exitCode=60 Oct 30 23:45:16.752610 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:16.752519 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq" event={"ID":"54e5ee1a-c524-4f52-bc9c-870411e7f17e","Type":"ContainerDied","Data":"4506846db209ea49c809183ab55de82d43986fe129c4f71bde61019dae70ed76"} Oct 30 23:45:16.752610 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:16.752561 4952 scope.go:122] "RemoveContainer" containerID="473f2cb4c743b2d9e4a416aea0321d98f2d6c53610c59bbd123052c110ea2381" Oct 30 23:45:16.752843 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:16.752818 4952 scope.go:122] "RemoveContainer" containerID="4506846db209ea49c809183ab55de82d43986fe129c4f71bde61019dae70ed76" Oct 30 23:45:16.753168 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:45:16.753147 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq_odh-ai-gateway-infra(54e5ee1a-c524-4f52-bc9c-870411e7f17e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq" podUID="54e5ee1a-c524-4f52-bc9c-870411e7f17e" Oct 30 23:45:16.869848 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:16.869824 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq"] Oct 30 23:45:16.963628 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:16.963591 4952 scope.go:122] "RemoveContainer" containerID="d1f240d2a77e59d5b558672b2fc3ff2d08ca64e8c7acd8c8429e229480065069" Oct 30 23:45:17.583875 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:17.583839 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:45:17.761387 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:17.761316 4952 generic.go:356] "Generic (PLEG): container finished" podID="032662ea-6f38-4092-8bda-d27d09daf88f" containerID="53e98f885e9bfa17cdd347b2275190fdcac91350d16cb2cd5a27ee2ff128a656" exitCode=60 Oct 30 23:45:17.761553 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:17.761383 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm" event={"ID":"032662ea-6f38-4092-8bda-d27d09daf88f","Type":"ContainerDied","Data":"53e98f885e9bfa17cdd347b2275190fdcac91350d16cb2cd5a27ee2ff128a656"} Oct 30 23:45:17.761553 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:17.761435 4952 scope.go:122] "RemoveContainer" containerID="70e36368b60f660664549cfc9281a7ae48a64e458cd8ededd6c93e4a041c5d01" Oct 30 23:45:17.761879 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:17.761857 4952 scope.go:122] "RemoveContainer" containerID="53e98f885e9bfa17cdd347b2275190fdcac91350d16cb2cd5a27ee2ff128a656" Oct 30 23:45:17.762407 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:45:17.762304 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm_odh-ai-gateway-infra(032662ea-6f38-4092-8bda-d27d09daf88f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm" podUID="032662ea-6f38-4092-8bda-d27d09daf88f" Oct 30 23:45:17.765661 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:17.765631 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m" event={"ID":"89959083-95eb-4626-b1b2-4b7ac9d76840","Type":"ContainerStarted","Data":"faa478f694b240d003c50bcb3d7b175f58880fbb869cbb9368c1422c82ffcf45"} Oct 30 23:45:17.875015 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:17.874981 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm"] Oct 30 23:45:17.940740 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:17.940712 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq" Oct 30 23:45:17.964200 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:17.964175 4952 scope.go:122] "RemoveContainer" containerID="4ff703cdb460a2b1bfa8a9b4b4bf3aed0025f53d385d76063b2737f41da7ea3b" Oct 30 23:45:17.994392 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:17.994367 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54e5ee1a-c524-4f52-bc9c-870411e7f17e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e5ee1a-c524-4f52-bc9c-870411e7f17e-maas-api-service-ca\") pod \"54e5ee1a-c524-4f52-bc9c-870411e7f17e\" (UID: \"54e5ee1a-c524-4f52-bc9c-870411e7f17e\") " Oct 30 23:45:17.994524 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:17.994509 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54e5ee1a-c524-4f52-bc9c-870411e7f17e-kube-api-access-2cxnq\" (UniqueName: \"kubernetes.io/projected/54e5ee1a-c524-4f52-bc9c-870411e7f17e-kube-api-access-2cxnq\") pod \"54e5ee1a-c524-4f52-bc9c-870411e7f17e\" (UID: \"54e5ee1a-c524-4f52-bc9c-870411e7f17e\") " Oct 30 23:45:17.994719 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:17.994698 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54e5ee1a-c524-4f52-bc9c-870411e7f17e-maas-api-service-ca" pod "54e5ee1a-c524-4f52-bc9c-870411e7f17e" (UID: "54e5ee1a-c524-4f52-bc9c-870411e7f17e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:45:17.994954 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:17.994940 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e5ee1a-c524-4f52-bc9c-870411e7f17e-maas-api-service-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:45:17.996638 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:17.996617 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54e5ee1a-c524-4f52-bc9c-870411e7f17e-kube-api-access-2cxnq" pod "54e5ee1a-c524-4f52-bc9c-870411e7f17e" (UID: "54e5ee1a-c524-4f52-bc9c-870411e7f17e"). InnerVolumeSpecName "kube-api-access-2cxnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:45:18.096666 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:18.096638 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cxnq\" (UniqueName: \"kubernetes.io/projected/54e5ee1a-c524-4f52-bc9c-870411e7f17e-kube-api-access-2cxnq\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:45:18.776393 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:18.776339 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq" event={"ID":"54e5ee1a-c524-4f52-bc9c-870411e7f17e","Type":"ContainerDied","Data":"acf68f8cc1f03601f055fd8e056d173609e9b85c99ec9abae8bfc5b23c4da3d8"} Oct 30 23:45:18.776547 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:18.776354 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq" Oct 30 23:45:18.776547 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:18.776413 4952 scope.go:122] "RemoveContainer" containerID="4506846db209ea49c809183ab55de82d43986fe129c4f71bde61019dae70ed76" Oct 30 23:45:18.782805 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:18.782780 4952 generic.go:356] "Generic (PLEG): container finished" podID="89959083-95eb-4626-b1b2-4b7ac9d76840" containerID="faa478f694b240d003c50bcb3d7b175f58880fbb869cbb9368c1422c82ffcf45" exitCode=60 Oct 30 23:45:18.782946 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:18.782854 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m" event={"ID":"89959083-95eb-4626-b1b2-4b7ac9d76840","Type":"ContainerDied","Data":"faa478f694b240d003c50bcb3d7b175f58880fbb869cbb9368c1422c82ffcf45"} Oct 30 23:45:18.783352 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:18.783327 4952 scope.go:122] "RemoveContainer" containerID="faa478f694b240d003c50bcb3d7b175f58880fbb869cbb9368c1422c82ffcf45" Oct 30 23:45:18.783733 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:45:18.783683 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m_odh-ai-gateway-infra(89959083-95eb-4626-b1b2-4b7ac9d76840)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m" podUID="89959083-95eb-4626-b1b2-4b7ac9d76840" Oct 30 23:45:18.794596 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:18.794565 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j" event={"ID":"5cd22a14-b583-4991-8ced-45c5b15919ba","Type":"ContainerStarted","Data":"b5719271f2c0c7808b736e58ce943d565143ff2a8d261ebc0c00c302df75944e"} Oct 30 23:45:18.802632 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:18.802577 4952 scope.go:122] "RemoveContainer" containerID="d1f240d2a77e59d5b558672b2fc3ff2d08ca64e8c7acd8c8429e229480065069" Oct 30 23:45:18.957315 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:18.957292 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm" Oct 30 23:45:19.009820 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:19.009789 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/032662ea-6f38-4092-8bda-d27d09daf88f-kube-api-access-xwv6l\" (UniqueName: \"kubernetes.io/projected/032662ea-6f38-4092-8bda-d27d09daf88f-kube-api-access-xwv6l\") pod \"032662ea-6f38-4092-8bda-d27d09daf88f\" (UID: \"032662ea-6f38-4092-8bda-d27d09daf88f\") " Oct 30 23:45:19.010289 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:19.009990 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/032662ea-6f38-4092-8bda-d27d09daf88f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/032662ea-6f38-4092-8bda-d27d09daf88f-maas-api-service-ca\") pod \"032662ea-6f38-4092-8bda-d27d09daf88f\" (UID: \"032662ea-6f38-4092-8bda-d27d09daf88f\") " Oct 30 23:45:19.010414 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:19.010372 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/032662ea-6f38-4092-8bda-d27d09daf88f-maas-api-service-ca" pod "032662ea-6f38-4092-8bda-d27d09daf88f" (UID: "032662ea-6f38-4092-8bda-d27d09daf88f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:45:19.010814 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:19.010783 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/032662ea-6f38-4092-8bda-d27d09daf88f-maas-api-service-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:45:19.016393 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:19.012670 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/032662ea-6f38-4092-8bda-d27d09daf88f-kube-api-access-xwv6l" pod "032662ea-6f38-4092-8bda-d27d09daf88f" (UID: "032662ea-6f38-4092-8bda-d27d09daf88f"). InnerVolumeSpecName "kube-api-access-xwv6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:45:19.016393 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:19.016314 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m"] Oct 30 23:45:19.020880 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:19.020860 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq"] Oct 30 23:45:19.035774 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:19.035719 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890065-f87kq"] Oct 30 23:45:19.111595 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:19.111567 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwv6l\" (UniqueName: \"kubernetes.io/projected/032662ea-6f38-4092-8bda-d27d09daf88f-kube-api-access-xwv6l\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:45:19.617389 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:19.617325 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:19.806353 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:19.806324 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm" Oct 30 23:45:19.806550 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:19.806372 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm" event={"ID":"032662ea-6f38-4092-8bda-d27d09daf88f","Type":"ContainerDied","Data":"4857a6b5d8b55f643c365cd5f2c1374cd842567955c192cec8370b3930116db8"} Oct 30 23:45:19.806550 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:19.806475 4952 scope.go:122] "RemoveContainer" containerID="53e98f885e9bfa17cdd347b2275190fdcac91350d16cb2cd5a27ee2ff128a656" Oct 30 23:45:19.812420 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:19.812391 4952 generic.go:356] "Generic (PLEG): container finished" podID="5cd22a14-b583-4991-8ced-45c5b15919ba" containerID="b5719271f2c0c7808b736e58ce943d565143ff2a8d261ebc0c00c302df75944e" exitCode=60 Oct 30 23:45:19.812619 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:19.812438 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j" event={"ID":"5cd22a14-b583-4991-8ced-45c5b15919ba","Type":"ContainerDied","Data":"b5719271f2c0c7808b736e58ce943d565143ff2a8d261ebc0c00c302df75944e"} Oct 30 23:45:19.813083 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:19.813018 4952 scope.go:122] "RemoveContainer" containerID="b5719271f2c0c7808b736e58ce943d565143ff2a8d261ebc0c00c302df75944e" Oct 30 23:45:19.813452 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:45:19.813419 4952 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j_odh-ai-gateway-infra(5cd22a14-b583-4991-8ced-45c5b15919ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j" podUID="5cd22a14-b583-4991-8ced-45c5b15919ba" Oct 30 23:45:19.823692 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:19.823670 4952 scope.go:122] "RemoveContainer" containerID="4ff703cdb460a2b1bfa8a9b4b4bf3aed0025f53d385d76063b2737f41da7ea3b" Oct 30 23:45:19.841858 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:19.841836 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:19.973139 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:19.973111 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm"] Oct 30 23:45:19.976354 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:19.976331 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b81a7a-29890065-7vdqm"] Oct 30 23:45:19.995692 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:19.995668 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m" Oct 30 23:45:20.123727 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:20.123695 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89959083-95eb-4626-b1b2-4b7ac9d76840-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89959083-95eb-4626-b1b2-4b7ac9d76840-maas-api-service-ca\") pod \"89959083-95eb-4626-b1b2-4b7ac9d76840\" (UID: \"89959083-95eb-4626-b1b2-4b7ac9d76840\") " Oct 30 23:45:20.124295 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:20.123884 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89959083-95eb-4626-b1b2-4b7ac9d76840-kube-api-access-8jtqb\" (UniqueName: \"kubernetes.io/projected/89959083-95eb-4626-b1b2-4b7ac9d76840-kube-api-access-8jtqb\") pod \"89959083-95eb-4626-b1b2-4b7ac9d76840\" (UID: \"89959083-95eb-4626-b1b2-4b7ac9d76840\") " Oct 30 23:45:20.124295 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:20.124109 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89959083-95eb-4626-b1b2-4b7ac9d76840-maas-api-service-ca" pod "89959083-95eb-4626-b1b2-4b7ac9d76840" (UID: "89959083-95eb-4626-b1b2-4b7ac9d76840"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:45:20.124486 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:20.124462 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89959083-95eb-4626-b1b2-4b7ac9d76840-maas-api-service-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:45:20.126452 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:20.126426 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89959083-95eb-4626-b1b2-4b7ac9d76840-kube-api-access-8jtqb" pod "89959083-95eb-4626-b1b2-4b7ac9d76840" (UID: "89959083-95eb-4626-b1b2-4b7ac9d76840"). InnerVolumeSpecName "kube-api-access-8jtqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:45:20.164230 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:20.164203 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j"] Oct 30 23:45:20.225593 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:20.225560 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jtqb\" (UniqueName: \"kubernetes.io/projected/89959083-95eb-4626-b1b2-4b7ac9d76840-kube-api-access-8jtqb\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:45:20.823335 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:20.823295 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m" Oct 30 23:45:20.825787 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:20.823303 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m" event={"ID":"89959083-95eb-4626-b1b2-4b7ac9d76840","Type":"ContainerDied","Data":"5149a838a32f312817638c9a60698e92f91b86e1d28b4074f5f03c36d23e9e46"} Oct 30 23:45:20.825787 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:20.823426 4952 scope.go:122] "RemoveContainer" containerID="faa478f694b240d003c50bcb3d7b175f58880fbb869cbb9368c1422c82ffcf45" Oct 30 23:45:20.972116 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:20.972086 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="032662ea-6f38-4092-8bda-d27d09daf88f" path="/var/lib/kubelet/pods/032662ea-6f38-4092-8bda-d27d09daf88f/volumes" Oct 30 23:45:20.972503 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:20.972489 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54e5ee1a-c524-4f52-bc9c-870411e7f17e" path="/var/lib/kubelet/pods/54e5ee1a-c524-4f52-bc9c-870411e7f17e/volumes" Oct 30 23:45:21.016370 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:21.016344 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j" Oct 30 23:45:21.106631 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:21.106595 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m"] Oct 30 23:45:21.111673 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:21.111645 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3be34b-29890065-85g9m"] Oct 30 23:45:21.140363 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:21.140322 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cd22a14-b583-4991-8ced-45c5b15919ba-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd22a14-b583-4991-8ced-45c5b15919ba-maas-api-service-ca\") pod \"5cd22a14-b583-4991-8ced-45c5b15919ba\" (UID: \"5cd22a14-b583-4991-8ced-45c5b15919ba\") " Oct 30 23:45:21.140838 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:21.140457 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cd22a14-b583-4991-8ced-45c5b15919ba-kube-api-access-sqgcj\" (UniqueName: \"kubernetes.io/projected/5cd22a14-b583-4991-8ced-45c5b15919ba-kube-api-access-sqgcj\") pod \"5cd22a14-b583-4991-8ced-45c5b15919ba\" (UID: \"5cd22a14-b583-4991-8ced-45c5b15919ba\") " Oct 30 23:45:21.141538 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:21.141488 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cd22a14-b583-4991-8ced-45c5b15919ba-maas-api-service-ca" pod "5cd22a14-b583-4991-8ced-45c5b15919ba" (UID: "5cd22a14-b583-4991-8ced-45c5b15919ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:45:21.143282 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:21.143256 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cd22a14-b583-4991-8ced-45c5b15919ba-kube-api-access-sqgcj" pod "5cd22a14-b583-4991-8ced-45c5b15919ba" (UID: "5cd22a14-b583-4991-8ced-45c5b15919ba"). InnerVolumeSpecName "kube-api-access-sqgcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:45:21.152360 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:45:21.152142 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:45:21.241786 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:21.241756 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd22a14-b583-4991-8ced-45c5b15919ba-maas-api-service-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:45:21.241786 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:21.241785 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqgcj\" (UniqueName: \"kubernetes.io/projected/5cd22a14-b583-4991-8ced-45c5b15919ba-kube-api-access-sqgcj\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:45:21.832807 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:21.832779 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j" Oct 30 23:45:21.832807 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:21.832794 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j" event={"ID":"5cd22a14-b583-4991-8ced-45c5b15919ba","Type":"ContainerDied","Data":"5377ecefe5f6e33b61957d3f22b33cc77f4e440bb42bc02399ef87bc72ffe464"} Oct 30 23:45:21.832936 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:21.832891 4952 scope.go:122] "RemoveContainer" containerID="b5719271f2c0c7808b736e58ce943d565143ff2a8d261ebc0c00c302df75944e" Oct 30 23:45:21.999814 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:21.999772 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j"] Oct 30 23:45:22.009199 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:22.009169 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890065-2s69j"] Oct 30 23:45:22.979193 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:22.979160 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cd22a14-b583-4991-8ced-45c5b15919ba" path="/var/lib/kubelet/pods/5cd22a14-b583-4991-8ced-45c5b15919ba/volumes" Oct 30 23:45:22.979792 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:22.979759 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89959083-95eb-4626-b1b2-4b7ac9d76840" path="/var/lib/kubelet/pods/89959083-95eb-4626-b1b2-4b7ac9d76840/volumes" Oct 30 23:45:23.956761 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:23.956728 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bcc619-775dbf957-vpn64"] Oct 30 23:45:23.957008 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:23.956968 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcc619-775dbf957-vpn64" podUID="4f85606f-6cb6-442e-a5e1-3a57fb732f93" containerName="maas-api" containerID="cri-o://820c25471f94e0f5786d795da0e58cf0c11194b5b3a5c40d321e0b34d571e6d6" gracePeriod=30 Oct 30 23:45:24.129162 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:24.129130 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:24.274175 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:24.274147 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcc619-775dbf957-vpn64" Oct 30 23:45:24.365096 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:24.365006 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:24.392745 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:24.392704 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f85606f-6cb6-442e-a5e1-3a57fb732f93-kube-api-access-t7qjk\" (UniqueName: \"kubernetes.io/projected/4f85606f-6cb6-442e-a5e1-3a57fb732f93-kube-api-access-t7qjk\") pod \"4f85606f-6cb6-442e-a5e1-3a57fb732f93\" (UID: \"4f85606f-6cb6-442e-a5e1-3a57fb732f93\") " Oct 30 23:45:24.393068 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:24.393021 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f85606f-6cb6-442e-a5e1-3a57fb732f93-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f85606f-6cb6-442e-a5e1-3a57fb732f93-metrics-certs\") pod \"4f85606f-6cb6-442e-a5e1-3a57fb732f93\" (UID: \"4f85606f-6cb6-442e-a5e1-3a57fb732f93\") " Oct 30 23:45:24.393279 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:24.393262 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f85606f-6cb6-442e-a5e1-3a57fb732f93-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f85606f-6cb6-442e-a5e1-3a57fb732f93-openshift-service-ca\") pod \"4f85606f-6cb6-442e-a5e1-3a57fb732f93\" (UID: \"4f85606f-6cb6-442e-a5e1-3a57fb732f93\") " Oct 30 23:45:24.393504 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:24.393486 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f85606f-6cb6-442e-a5e1-3a57fb732f93-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f85606f-6cb6-442e-a5e1-3a57fb732f93-maas-api-tls\") pod \"4f85606f-6cb6-442e-a5e1-3a57fb732f93\" (UID: \"4f85606f-6cb6-442e-a5e1-3a57fb732f93\") " Oct 30 23:45:24.400728 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:24.400700 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f85606f-6cb6-442e-a5e1-3a57fb732f93-openshift-service-ca" pod "4f85606f-6cb6-442e-a5e1-3a57fb732f93" (UID: "4f85606f-6cb6-442e-a5e1-3a57fb732f93"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:45:24.411249 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:24.411178 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f85606f-6cb6-442e-a5e1-3a57fb732f93-maas-api-tls" pod "4f85606f-6cb6-442e-a5e1-3a57fb732f93" (UID: "4f85606f-6cb6-442e-a5e1-3a57fb732f93"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:45:24.413188 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:24.413164 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f85606f-6cb6-442e-a5e1-3a57fb732f93-metrics-certs" pod "4f85606f-6cb6-442e-a5e1-3a57fb732f93" (UID: "4f85606f-6cb6-442e-a5e1-3a57fb732f93"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:45:24.450468 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:24.450412 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f85606f-6cb6-442e-a5e1-3a57fb732f93-kube-api-access-t7qjk" pod "4f85606f-6cb6-442e-a5e1-3a57fb732f93" (UID: "4f85606f-6cb6-442e-a5e1-3a57fb732f93"). InnerVolumeSpecName "kube-api-access-t7qjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:45:24.495826 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:24.495752 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f85606f-6cb6-442e-a5e1-3a57fb732f93-maas-api-tls\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:45:24.495826 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:24.495787 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7qjk\" (UniqueName: \"kubernetes.io/projected/4f85606f-6cb6-442e-a5e1-3a57fb732f93-kube-api-access-t7qjk\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:45:24.495826 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:24.495801 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f85606f-6cb6-442e-a5e1-3a57fb732f93-metrics-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:45:24.495826 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:24.495815 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f85606f-6cb6-442e-a5e1-3a57fb732f93-openshift-service-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:45:24.886708 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:24.886672 4952 generic.go:356] "Generic (PLEG): container finished" podID="4f85606f-6cb6-442e-a5e1-3a57fb732f93" containerID="820c25471f94e0f5786d795da0e58cf0c11194b5b3a5c40d321e0b34d571e6d6" exitCode=0 Oct 30 23:45:24.886927 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:24.886760 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcc619-775dbf957-vpn64" event={"ID":"4f85606f-6cb6-442e-a5e1-3a57fb732f93","Type":"ContainerDied","Data":"820c25471f94e0f5786d795da0e58cf0c11194b5b3a5c40d321e0b34d571e6d6"} Oct 30 23:45:24.886927 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:24.886813 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcc619-775dbf957-vpn64" event={"ID":"4f85606f-6cb6-442e-a5e1-3a57fb732f93","Type":"ContainerDied","Data":"34589a869082fed513fe45e7d7d69c62853eb651504d73011d6a33f62622b05d"} Oct 30 23:45:24.886927 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:24.886845 4952 scope.go:122] "RemoveContainer" containerID="820c25471f94e0f5786d795da0e58cf0c11194b5b3a5c40d321e0b34d571e6d6" Oct 30 23:45:24.887147 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:24.886956 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcc619-775dbf957-vpn64" Oct 30 23:45:24.931501 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:24.928292 4952 scope.go:122] "RemoveContainer" containerID="820c25471f94e0f5786d795da0e58cf0c11194b5b3a5c40d321e0b34d571e6d6" Oct 30 23:45:24.931802 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:45:24.931691 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"820c25471f94e0f5786d795da0e58cf0c11194b5b3a5c40d321e0b34d571e6d6\": container with ID starting with 820c25471f94e0f5786d795da0e58cf0c11194b5b3a5c40d321e0b34d571e6d6 not found: ID does not exist" containerID="820c25471f94e0f5786d795da0e58cf0c11194b5b3a5c40d321e0b34d571e6d6" Oct 30 23:45:24.931802 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:24.931730 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"820c25471f94e0f5786d795da0e58cf0c11194b5b3a5c40d321e0b34d571e6d6"} err="failed to get container status \"820c25471f94e0f5786d795da0e58cf0c11194b5b3a5c40d321e0b34d571e6d6\": rpc error: code = NotFound desc = could not find container \"820c25471f94e0f5786d795da0e58cf0c11194b5b3a5c40d321e0b34d571e6d6\": container with ID starting with 820c25471f94e0f5786d795da0e58cf0c11194b5b3a5c40d321e0b34d571e6d6 not found: ID does not exist" Oct 30 23:45:25.059345 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:25.059312 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bcc619-775dbf957-vpn64"] Oct 30 23:45:25.109479 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:25.109448 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bcc619-775dbf957-vpn64"] Oct 30 23:45:25.642505 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:25.642475 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:25.973562 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:25.973474 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:26.194436 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:26.194397 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:26.619833 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:26.619803 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:26.971214 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:26.971144 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f85606f-6cb6-442e-a5e1-3a57fb732f93" path="/var/lib/kubelet/pods/4f85606f-6cb6-442e-a5e1-3a57fb732f93/volumes" Oct 30 23:45:29.908842 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:29.908805 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:31.277355 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:31.277324 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:31.577948 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:31.577869 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:32.666645 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:32.666602 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:32.905969 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:32.905930 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:33.136303 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:33.136269 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:36.586313 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:36.586272 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:41.328555 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:41.328516 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:41.559342 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:41.559152 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:42.475572 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:42.475534 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:42.920464 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:42.920434 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:45.359058 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:45.359010 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-26ce6b-6964c5f8b7-xmfd7"] Oct 30 23:45:45.712966 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:45.712875 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:46.002395 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:46.002304 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:46.303195 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:46.303111 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:48.856679 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:48.856646 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:50.360294 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:50.360256 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26ce6b-6964c5f8b7-xmfd7" podUID="3b443f22-9bac-4a46-9b4a-8982e0006fc4" containerName="payload-pre-processing" containerID="cri-o://93aed4ffe1c2fb328cef0b48b3c6dfe2306fb367535f63d33d854904f630b027" gracePeriod=25 Oct 30 23:45:50.634122 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:50.634098 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26ce6b-6964c5f8b7-xmfd7" Oct 30 23:45:50.755726 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:50.755694 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b443f22-9bac-4a46-9b4a-8982e0006fc4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b443f22-9bac-4a46-9b4a-8982e0006fc4-plugins-config-volume\") pod \"3b443f22-9bac-4a46-9b4a-8982e0006fc4\" (UID: \"3b443f22-9bac-4a46-9b4a-8982e0006fc4\") " Oct 30 23:45:50.755936 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:50.755918 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b443f22-9bac-4a46-9b4a-8982e0006fc4-kube-api-access-mxp49\" (UniqueName: \"kubernetes.io/projected/3b443f22-9bac-4a46-9b4a-8982e0006fc4-kube-api-access-mxp49\") pod \"3b443f22-9bac-4a46-9b4a-8982e0006fc4\" (UID: \"3b443f22-9bac-4a46-9b4a-8982e0006fc4\") " Oct 30 23:45:50.756499 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:50.756231 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b443f22-9bac-4a46-9b4a-8982e0006fc4-plugins-config-volume" pod "3b443f22-9bac-4a46-9b4a-8982e0006fc4" (UID: "3b443f22-9bac-4a46-9b4a-8982e0006fc4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:45:50.756499 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:50.756445 4952 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b443f22-9bac-4a46-9b4a-8982e0006fc4-plugins-config-volume\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:45:50.760315 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:50.760287 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b443f22-9bac-4a46-9b4a-8982e0006fc4-kube-api-access-mxp49" pod "3b443f22-9bac-4a46-9b4a-8982e0006fc4" (UID: "3b443f22-9bac-4a46-9b4a-8982e0006fc4"). InnerVolumeSpecName "kube-api-access-mxp49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:45:50.858123 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:50.858090 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxp49\" (UniqueName: \"kubernetes.io/projected/3b443f22-9bac-4a46-9b4a-8982e0006fc4-kube-api-access-mxp49\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:45:51.084072 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:51.084024 4952 generic.go:356] "Generic (PLEG): container finished" podID="3b443f22-9bac-4a46-9b4a-8982e0006fc4" containerID="93aed4ffe1c2fb328cef0b48b3c6dfe2306fb367535f63d33d854904f630b027" exitCode=0 Oct 30 23:45:51.084206 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:51.084092 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26ce6b-6964c5f8b7-xmfd7" Oct 30 23:45:51.084206 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:51.084088 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26ce6b-6964c5f8b7-xmfd7" event={"ID":"3b443f22-9bac-4a46-9b4a-8982e0006fc4","Type":"ContainerDied","Data":"93aed4ffe1c2fb328cef0b48b3c6dfe2306fb367535f63d33d854904f630b027"} Oct 30 23:45:51.084286 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:51.084209 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-26ce6b-6964c5f8b7-xmfd7" event={"ID":"3b443f22-9bac-4a46-9b4a-8982e0006fc4","Type":"ContainerDied","Data":"368fb23d3b83bc40a729ef401e95e5199b6dba3e0409631f3e4f535686978f4e"} Oct 30 23:45:51.084286 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:51.084235 4952 scope.go:122] "RemoveContainer" containerID="93aed4ffe1c2fb328cef0b48b3c6dfe2306fb367535f63d33d854904f630b027" Oct 30 23:45:51.094822 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:51.094802 4952 scope.go:122] "RemoveContainer" containerID="93aed4ffe1c2fb328cef0b48b3c6dfe2306fb367535f63d33d854904f630b027" Oct 30 23:45:51.095089 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:45:51.095072 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93aed4ffe1c2fb328cef0b48b3c6dfe2306fb367535f63d33d854904f630b027\": container with ID starting with 93aed4ffe1c2fb328cef0b48b3c6dfe2306fb367535f63d33d854904f630b027 not found: ID does not exist" containerID="93aed4ffe1c2fb328cef0b48b3c6dfe2306fb367535f63d33d854904f630b027" Oct 30 23:45:51.095145 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:51.095097 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93aed4ffe1c2fb328cef0b48b3c6dfe2306fb367535f63d33d854904f630b027"} err="failed to get container status \"93aed4ffe1c2fb328cef0b48b3c6dfe2306fb367535f63d33d854904f630b027\": rpc error: code = NotFound desc = could not find container \"93aed4ffe1c2fb328cef0b48b3c6dfe2306fb367535f63d33d854904f630b027\": container with ID starting with 93aed4ffe1c2fb328cef0b48b3c6dfe2306fb367535f63d33d854904f630b027 not found: ID does not exist" Oct 30 23:45:51.161734 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:45:51.161694 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:45:51.178029 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:51.177988 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:51.251397 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:51.251328 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-26ce6b-6964c5f8b7-xmfd7"] Oct 30 23:45:51.255104 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:51.255082 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-26ce6b-6964c5f8b7-xmfd7"] Oct 30 23:45:52.970993 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:52.970950 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b443f22-9bac-4a46-9b4a-8982e0006fc4" path="/var/lib/kubelet/pods/3b443f22-9bac-4a46-9b4a-8982e0006fc4/volumes" Oct 30 23:45:55.962185 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:55.962125 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:45:59.942080 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:59.941990 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6"] Oct 30 23:45:59.942510 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:45:59.942194 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6" podUID="f90492f3-9a0a-447d-8a29-6a91e026a5b1" containerName="maas-api" containerID="cri-o://2021a6717a0e0e7cf748a5eca59cbe799e8e372a1a472b3e54d81c247cebb4d5" gracePeriod=30 Oct 30 23:46:00.187364 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:00.186915 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:00.201491 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:00.201406 4952 generic.go:356] "Generic (PLEG): container finished" podID="f90492f3-9a0a-447d-8a29-6a91e026a5b1" containerID="2021a6717a0e0e7cf748a5eca59cbe799e8e372a1a472b3e54d81c247cebb4d5" exitCode=0 Oct 30 23:46:00.201975 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:00.201926 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6" event={"ID":"f90492f3-9a0a-447d-8a29-6a91e026a5b1","Type":"ContainerDied","Data":"2021a6717a0e0e7cf748a5eca59cbe799e8e372a1a472b3e54d81c247cebb4d5"} Oct 30 23:46:00.283843 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:00.283816 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6" Oct 30 23:46:00.390748 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:00.390709 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f90492f3-9a0a-447d-8a29-6a91e026a5b1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f90492f3-9a0a-447d-8a29-6a91e026a5b1-openshift-service-ca\") pod \"f90492f3-9a0a-447d-8a29-6a91e026a5b1\" (UID: \"f90492f3-9a0a-447d-8a29-6a91e026a5b1\") " Oct 30 23:46:00.390928 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:00.390869 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f90492f3-9a0a-447d-8a29-6a91e026a5b1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f90492f3-9a0a-447d-8a29-6a91e026a5b1-maas-api-tls\") pod \"f90492f3-9a0a-447d-8a29-6a91e026a5b1\" (UID: \"f90492f3-9a0a-447d-8a29-6a91e026a5b1\") " Oct 30 23:46:00.391137 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:00.391113 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f90492f3-9a0a-447d-8a29-6a91e026a5b1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f90492f3-9a0a-447d-8a29-6a91e026a5b1-metrics-certs\") pod \"f90492f3-9a0a-447d-8a29-6a91e026a5b1\" (UID: \"f90492f3-9a0a-447d-8a29-6a91e026a5b1\") " Oct 30 23:46:00.391234 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:00.391128 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f90492f3-9a0a-447d-8a29-6a91e026a5b1-openshift-service-ca" pod "f90492f3-9a0a-447d-8a29-6a91e026a5b1" (UID: "f90492f3-9a0a-447d-8a29-6a91e026a5b1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:46:00.391234 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:00.391160 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f90492f3-9a0a-447d-8a29-6a91e026a5b1-kube-api-access-rljbz\" (UniqueName: \"kubernetes.io/projected/f90492f3-9a0a-447d-8a29-6a91e026a5b1-kube-api-access-rljbz\") pod \"f90492f3-9a0a-447d-8a29-6a91e026a5b1\" (UID: \"f90492f3-9a0a-447d-8a29-6a91e026a5b1\") " Oct 30 23:46:00.391580 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:00.391556 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f90492f3-9a0a-447d-8a29-6a91e026a5b1-openshift-service-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:00.393547 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:00.393522 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f90492f3-9a0a-447d-8a29-6a91e026a5b1-metrics-certs" pod "f90492f3-9a0a-447d-8a29-6a91e026a5b1" (UID: "f90492f3-9a0a-447d-8a29-6a91e026a5b1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:46:00.394015 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:00.393987 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f90492f3-9a0a-447d-8a29-6a91e026a5b1-maas-api-tls" pod "f90492f3-9a0a-447d-8a29-6a91e026a5b1" (UID: "f90492f3-9a0a-447d-8a29-6a91e026a5b1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:46:00.394015 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:00.394010 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f90492f3-9a0a-447d-8a29-6a91e026a5b1-kube-api-access-rljbz" pod "f90492f3-9a0a-447d-8a29-6a91e026a5b1" (UID: "f90492f3-9a0a-447d-8a29-6a91e026a5b1"). InnerVolumeSpecName "kube-api-access-rljbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:46:00.493359 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:00.493284 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rljbz\" (UniqueName: \"kubernetes.io/projected/f90492f3-9a0a-447d-8a29-6a91e026a5b1-kube-api-access-rljbz\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:00.493359 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:00.493312 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f90492f3-9a0a-447d-8a29-6a91e026a5b1-maas-api-tls\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:00.493359 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:00.493324 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f90492f3-9a0a-447d-8a29-6a91e026a5b1-metrics-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:01.209491 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:01.209458 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6" Oct 30 23:46:01.209916 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:01.209490 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6" event={"ID":"f90492f3-9a0a-447d-8a29-6a91e026a5b1","Type":"ContainerDied","Data":"92c0f3fb2e15a01d955adabea84c55192779d88c3b74a23ca53eff16b9d38268"} Oct 30 23:46:01.209916 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:01.209551 4952 scope.go:122] "RemoveContainer" containerID="2021a6717a0e0e7cf748a5eca59cbe799e8e372a1a472b3e54d81c247cebb4d5" Oct 30 23:46:01.374332 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:01.374304 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6"] Oct 30 23:46:01.423998 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:01.423969 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-989a4f-8564696f56-6pnb6"] Oct 30 23:46:01.648478 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:01.648391 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:01.867447 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:01.867413 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:02.090360 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:02.090328 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:02.578690 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:02.578596 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:02.971210 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:02.971176 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f90492f3-9a0a-447d-8a29-6a91e026a5b1" path="/var/lib/kubelet/pods/f90492f3-9a0a-447d-8a29-6a91e026a5b1/volumes" Oct 30 23:46:06.982116 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:06.982082 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:09.003885 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:09.003848 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:09.226348 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:09.226309 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:09.246886 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:09.246859 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t"] Oct 30 23:46:09.247144 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:09.247120 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" podUID="ec9e7fff-1e25-4f16-ab95-f1a1d65448da" containerName="istio-proxy" containerID="cri-o://59385393f120d44716737047ed32fab568f953953c10981e08ab28b4dcc48fbe" gracePeriod=30 Oct 30 23:46:09.448052 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:09.448005 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:11.873665 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:11.873633 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:12.217928 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:12.217855 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:12.520860 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:12.520780 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:13.173135 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:13.173082 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:13.420457 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:13.420419 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2"] Oct 30 23:46:13.420714 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:13.420677 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" podUID="a365be8c-838f-46e0-a439-00700619a558" containerName="istio-proxy" containerID="cri-o://7f377bc9dec227a8d54c09e6ab0cc1bd3b94ffd7db12f3626d7ddf130245db28" gracePeriod=30 Oct 30 23:46:13.426299 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:13.426224 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:13.451162 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:13.451127 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w"] Oct 30 23:46:13.451442 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:13.451402 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" podUID="d22e1d9f-31a2-43a0-999c-084069d6a9eb" containerName="istio-proxy" containerID="cri-o://18cc3d0fddc8a5f7c62694b0eabd1e51887acdbdcf4e8656a8d7eff92f6e6155" gracePeriod=30 Oct 30 23:46:13.455893 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:13.455871 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c"] Oct 30 23:46:13.456070 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:13.456028 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" podUID="3e71bd07-c0e7-4013-ac2d-67206a09b3e2" containerName="istio-proxy" containerID="cri-o://9f09a8cce7ab8093e99963d1f1f7688450135ec13a984d3057ad6023723b812a" gracePeriod=30 Oct 30 23:46:13.661187 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:13.661144 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:14.465075 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:14.464492 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:14.525139 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:14.525078 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" Oct 30 23:46:14.671520 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:14.671489 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-istio-podinfo\") pod \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " Oct 30 23:46:14.671692 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:14.671554 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-istio-token\" (UniqueName: \"kubernetes.io/projected/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-istio-token\") pod \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " Oct 30 23:46:14.671692 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:14.671603 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-workload-certs\") pod \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " Oct 30 23:46:14.671692 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:14.671659 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-credential-socket\") pod \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " Oct 30 23:46:14.671879 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:14.671732 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-istio-data\") pod \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " Oct 30 23:46:14.671879 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:14.671785 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-workload-socket\") pod \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " Oct 30 23:46:14.671879 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:14.671825 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-kube-api-access-gd5kn\" (UniqueName: \"kubernetes.io/projected/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-kube-api-access-gd5kn\") pod \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " Oct 30 23:46:14.672075 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:14.671929 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-istio-envoy\") pod \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " Oct 30 23:46:14.672075 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:14.671973 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-istiod-ca-cert\") pod \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\" (UID: \"ec9e7fff-1e25-4f16-ab95-f1a1d65448da\") " Oct 30 23:46:14.672075 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:14.671980 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-workload-certs" pod "ec9e7fff-1e25-4f16-ab95-f1a1d65448da" (UID: "ec9e7fff-1e25-4f16-ab95-f1a1d65448da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:46:14.672284 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:14.672256 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-istio-data" pod "ec9e7fff-1e25-4f16-ab95-f1a1d65448da" (UID: "ec9e7fff-1e25-4f16-ab95-f1a1d65448da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:46:14.672614 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:14.672587 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-istiod-ca-cert" pod "ec9e7fff-1e25-4f16-ab95-f1a1d65448da" (UID: "ec9e7fff-1e25-4f16-ab95-f1a1d65448da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:46:14.673122 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:14.672687 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-workload-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:14.673122 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:14.673018 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-istio-data\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:14.673122 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:14.672840 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-workload-socket" pod "ec9e7fff-1e25-4f16-ab95-f1a1d65448da" (UID: "ec9e7fff-1e25-4f16-ab95-f1a1d65448da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:46:14.673122 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:14.672949 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-credential-socket" pod "ec9e7fff-1e25-4f16-ab95-f1a1d65448da" (UID: "ec9e7fff-1e25-4f16-ab95-f1a1d65448da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:46:14.674363 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:14.674337 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-istio-podinfo" pod "ec9e7fff-1e25-4f16-ab95-f1a1d65448da" (UID: "ec9e7fff-1e25-4f16-ab95-f1a1d65448da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:46:14.674670 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:14.674650 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-istio-token" pod "ec9e7fff-1e25-4f16-ab95-f1a1d65448da" (UID: "ec9e7fff-1e25-4f16-ab95-f1a1d65448da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:46:14.674953 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:14.674935 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-kube-api-access-gd5kn" pod "ec9e7fff-1e25-4f16-ab95-f1a1d65448da" (UID: "ec9e7fff-1e25-4f16-ab95-f1a1d65448da"). InnerVolumeSpecName "kube-api-access-gd5kn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:46:14.675288 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:14.675270 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-istio-envoy" pod "ec9e7fff-1e25-4f16-ab95-f1a1d65448da" (UID: "ec9e7fff-1e25-4f16-ab95-f1a1d65448da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:46:14.690795 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:14.690773 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:14.774256 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:14.774222 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-credential-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:14.774256 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:14.774248 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-workload-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:14.774256 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:14.774258 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd5kn\" (UniqueName: \"kubernetes.io/projected/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-kube-api-access-gd5kn\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:14.774456 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:14.774269 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-istio-envoy\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:14.774456 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:14.774278 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-istiod-ca-cert\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:14.774456 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:14.774288 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-istio-podinfo\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:14.774456 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:14.774298 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec9e7fff-1e25-4f16-ab95-f1a1d65448da-istio-token\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:15.305736 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:15.305653 4952 generic.go:356] "Generic (PLEG): container finished" podID="ec9e7fff-1e25-4f16-ab95-f1a1d65448da" containerID="59385393f120d44716737047ed32fab568f953953c10981e08ab28b4dcc48fbe" exitCode=0 Oct 30 23:46:15.305736 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:15.305701 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" Oct 30 23:46:15.305937 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:15.305738 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" event={"ID":"ec9e7fff-1e25-4f16-ab95-f1a1d65448da","Type":"ContainerDied","Data":"59385393f120d44716737047ed32fab568f953953c10981e08ab28b4dcc48fbe"} Oct 30 23:46:15.305937 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:15.305794 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t" event={"ID":"ec9e7fff-1e25-4f16-ab95-f1a1d65448da","Type":"ContainerDied","Data":"4c099a90ac841770291f2a75ee197c79a5f94c8fee29f91109ff64b7c7fa9107"} Oct 30 23:46:15.305937 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:15.305824 4952 scope.go:122] "RemoveContainer" containerID="59385393f120d44716737047ed32fab568f953953c10981e08ab28b4dcc48fbe" Oct 30 23:46:15.316655 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:15.316636 4952 scope.go:122] "RemoveContainer" containerID="59385393f120d44716737047ed32fab568f953953c10981e08ab28b4dcc48fbe" Oct 30 23:46:15.316979 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:46:15.316956 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59385393f120d44716737047ed32fab568f953953c10981e08ab28b4dcc48fbe\": container with ID starting with 59385393f120d44716737047ed32fab568f953953c10981e08ab28b4dcc48fbe not found: ID does not exist" containerID="59385393f120d44716737047ed32fab568f953953c10981e08ab28b4dcc48fbe" Oct 30 23:46:15.317064 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:15.316987 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59385393f120d44716737047ed32fab568f953953c10981e08ab28b4dcc48fbe"} err="failed to get container status \"59385393f120d44716737047ed32fab568f953953c10981e08ab28b4dcc48fbe\": rpc error: code = NotFound desc = could not find container \"59385393f120d44716737047ed32fab568f953953c10981e08ab28b4dcc48fbe\": container with ID starting with 59385393f120d44716737047ed32fab568f953953c10981e08ab28b4dcc48fbe not found: ID does not exist" Oct 30 23:46:15.530379 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:15.530348 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t"] Oct 30 23:46:15.580102 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:15.580017 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-8pl8t"] Oct 30 23:46:15.943997 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:15.943962 4952 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-785d2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 23:46:15.944177 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:15.944018 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" podUID="a365be8c-838f-46e0-a439-00700619a558" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:46:16.775241 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:16.775203 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:16.970728 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:16.970693 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec9e7fff-1e25-4f16-ab95-f1a1d65448da" path="/var/lib/kubelet/pods/ec9e7fff-1e25-4f16-ab95-f1a1d65448da/volumes" Oct 30 23:46:18.726907 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.726877 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" Oct 30 23:46:18.764515 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.764486 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" Oct 30 23:46:18.785228 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.785209 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" Oct 30 23:46:18.816684 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.816662 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d22e1d9f-31a2-43a0-999c-084069d6a9eb-istio-token\" (UniqueName: \"kubernetes.io/projected/d22e1d9f-31a2-43a0-999c-084069d6a9eb-istio-token\") pod \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " Oct 30 23:46:18.816779 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.816709 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a365be8c-838f-46e0-a439-00700619a558-istio-token\" (UniqueName: \"kubernetes.io/projected/a365be8c-838f-46e0-a439-00700619a558-istio-token\") pod \"a365be8c-838f-46e0-a439-00700619a558\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " Oct 30 23:46:18.816779 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.816747 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-istio-data\") pod \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " Oct 30 23:46:18.816779 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.816768 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-workload-certs\") pod \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " Oct 30 23:46:18.816955 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.816830 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a365be8c-838f-46e0-a439-00700619a558-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a365be8c-838f-46e0-a439-00700619a558-istio-podinfo\") pod \"a365be8c-838f-46e0-a439-00700619a558\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " Oct 30 23:46:18.816955 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.816867 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a365be8c-838f-46e0-a439-00700619a558-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a365be8c-838f-46e0-a439-00700619a558-istiod-ca-cert\") pod \"a365be8c-838f-46e0-a439-00700619a558\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " Oct 30 23:46:18.816955 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.816912 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d22e1d9f-31a2-43a0-999c-084069d6a9eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d22e1d9f-31a2-43a0-999c-084069d6a9eb-credential-socket\") pod \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " Oct 30 23:46:18.816955 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.816945 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-workload-socket\") pod \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " Oct 30 23:46:18.817229 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.817001 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a365be8c-838f-46e0-a439-00700619a558-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a365be8c-838f-46e0-a439-00700619a558-credential-socket\") pod \"a365be8c-838f-46e0-a439-00700619a558\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " Oct 30 23:46:18.817229 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.817026 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-istio-data" pod "3e71bd07-c0e7-4013-ac2d-67206a09b3e2" (UID: "3e71bd07-c0e7-4013-ac2d-67206a09b3e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:46:18.817229 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.817071 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d22e1d9f-31a2-43a0-999c-084069d6a9eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d22e1d9f-31a2-43a0-999c-084069d6a9eb-istio-podinfo\") pod \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " Oct 30 23:46:18.817229 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.817167 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a365be8c-838f-46e0-a439-00700619a558-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a365be8c-838f-46e0-a439-00700619a558-workload-socket\") pod \"a365be8c-838f-46e0-a439-00700619a558\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " Oct 30 23:46:18.817229 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.817227 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-istio-token\" (UniqueName: \"kubernetes.io/projected/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-istio-token\") pod \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " Oct 30 23:46:18.817548 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.817288 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a365be8c-838f-46e0-a439-00700619a558-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a365be8c-838f-46e0-a439-00700619a558-workload-certs\") pod \"a365be8c-838f-46e0-a439-00700619a558\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " Oct 30 23:46:18.817548 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.817317 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-istiod-ca-cert\") pod \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " Oct 30 23:46:18.817548 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.817322 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d22e1d9f-31a2-43a0-999c-084069d6a9eb-credential-socket" pod "d22e1d9f-31a2-43a0-999c-084069d6a9eb" (UID: "d22e1d9f-31a2-43a0-999c-084069d6a9eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:46:18.817548 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.817356 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-istio-podinfo\") pod \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " Oct 30 23:46:18.817548 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.817488 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a365be8c-838f-46e0-a439-00700619a558-workload-socket" pod "a365be8c-838f-46e0-a439-00700619a558" (UID: "a365be8c-838f-46e0-a439-00700619a558"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:46:18.817854 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.817554 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-kube-api-access-b5hvv\" (UniqueName: \"kubernetes.io/projected/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-kube-api-access-b5hvv\") pod \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " Oct 30 23:46:18.817854 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.817583 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a365be8c-838f-46e0-a439-00700619a558-kube-api-access-w9f55\" (UniqueName: \"kubernetes.io/projected/a365be8c-838f-46e0-a439-00700619a558-kube-api-access-w9f55\") pod \"a365be8c-838f-46e0-a439-00700619a558\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " Oct 30 23:46:18.817854 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.817611 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d22e1d9f-31a2-43a0-999c-084069d6a9eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d22e1d9f-31a2-43a0-999c-084069d6a9eb-istiod-ca-cert\") pod \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " Oct 30 23:46:18.817854 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.817642 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a365be8c-838f-46e0-a439-00700619a558-istiod-ca-cert" pod "a365be8c-838f-46e0-a439-00700619a558" (UID: "a365be8c-838f-46e0-a439-00700619a558"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:46:18.817854 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.817676 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d22e1d9f-31a2-43a0-999c-084069d6a9eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d22e1d9f-31a2-43a0-999c-084069d6a9eb-workload-certs\") pod \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " Oct 30 23:46:18.817854 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.817707 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-istio-envoy\") pod \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " Oct 30 23:46:18.817854 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.817748 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a365be8c-838f-46e0-a439-00700619a558-workload-certs" pod "a365be8c-838f-46e0-a439-00700619a558" (UID: "a365be8c-838f-46e0-a439-00700619a558"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:46:18.817854 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.817772 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d22e1d9f-31a2-43a0-999c-084069d6a9eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d22e1d9f-31a2-43a0-999c-084069d6a9eb-istio-envoy\") pod \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " Oct 30 23:46:18.817854 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.817835 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d22e1d9f-31a2-43a0-999c-084069d6a9eb-kube-api-access-h4wk2\" (UniqueName: \"kubernetes.io/projected/d22e1d9f-31a2-43a0-999c-084069d6a9eb-kube-api-access-h4wk2\") pod \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " Oct 30 23:46:18.818349 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.817863 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d22e1d9f-31a2-43a0-999c-084069d6a9eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d22e1d9f-31a2-43a0-999c-084069d6a9eb-istio-data\") pod \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " Oct 30 23:46:18.818349 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.817899 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a365be8c-838f-46e0-a439-00700619a558-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a365be8c-838f-46e0-a439-00700619a558-istio-envoy\") pod \"a365be8c-838f-46e0-a439-00700619a558\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " Oct 30 23:46:18.818349 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.817929 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d22e1d9f-31a2-43a0-999c-084069d6a9eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d22e1d9f-31a2-43a0-999c-084069d6a9eb-workload-socket\") pod \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\" (UID: \"d22e1d9f-31a2-43a0-999c-084069d6a9eb\") " Oct 30 23:46:18.818349 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.817962 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a365be8c-838f-46e0-a439-00700619a558-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a365be8c-838f-46e0-a439-00700619a558-istio-data\") pod \"a365be8c-838f-46e0-a439-00700619a558\" (UID: \"a365be8c-838f-46e0-a439-00700619a558\") " Oct 30 23:46:18.818349 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.818031 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-credential-socket\") pod \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\" (UID: \"3e71bd07-c0e7-4013-ac2d-67206a09b3e2\") " Oct 30 23:46:18.818694 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.818663 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-istio-data\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:18.818778 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.818693 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a365be8c-838f-46e0-a439-00700619a558-istiod-ca-cert\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:18.818778 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.818711 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d22e1d9f-31a2-43a0-999c-084069d6a9eb-credential-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:18.818778 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.818726 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a365be8c-838f-46e0-a439-00700619a558-workload-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:18.818778 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.818742 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a365be8c-838f-46e0-a439-00700619a558-workload-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:18.820286 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.817966 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-workload-socket" pod "3e71bd07-c0e7-4013-ac2d-67206a09b3e2" (UID: "3e71bd07-c0e7-4013-ac2d-67206a09b3e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:46:18.820286 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.818156 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d22e1d9f-31a2-43a0-999c-084069d6a9eb-workload-certs" pod "d22e1d9f-31a2-43a0-999c-084069d6a9eb" (UID: "d22e1d9f-31a2-43a0-999c-084069d6a9eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:46:18.820286 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.818419 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-istiod-ca-cert" pod "3e71bd07-c0e7-4013-ac2d-67206a09b3e2" (UID: "3e71bd07-c0e7-4013-ac2d-67206a09b3e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:46:18.820286 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.818944 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-credential-socket" pod "3e71bd07-c0e7-4013-ac2d-67206a09b3e2" (UID: "3e71bd07-c0e7-4013-ac2d-67206a09b3e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:46:18.820286 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.819175 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a365be8c-838f-46e0-a439-00700619a558-credential-socket" pod "a365be8c-838f-46e0-a439-00700619a558" (UID: "a365be8c-838f-46e0-a439-00700619a558"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:46:18.820286 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.819459 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d22e1d9f-31a2-43a0-999c-084069d6a9eb-istiod-ca-cert" pod "d22e1d9f-31a2-43a0-999c-084069d6a9eb" (UID: "d22e1d9f-31a2-43a0-999c-084069d6a9eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:46:18.820286 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.819630 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-workload-certs" pod "3e71bd07-c0e7-4013-ac2d-67206a09b3e2" (UID: "3e71bd07-c0e7-4013-ac2d-67206a09b3e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:46:18.820286 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.819846 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d22e1d9f-31a2-43a0-999c-084069d6a9eb-workload-socket" pod "d22e1d9f-31a2-43a0-999c-084069d6a9eb" (UID: "d22e1d9f-31a2-43a0-999c-084069d6a9eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:46:18.820286 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.819747 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d22e1d9f-31a2-43a0-999c-084069d6a9eb-istio-podinfo" pod "d22e1d9f-31a2-43a0-999c-084069d6a9eb" (UID: "d22e1d9f-31a2-43a0-999c-084069d6a9eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:46:18.820286 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.820078 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a365be8c-838f-46e0-a439-00700619a558-istio-data" pod "a365be8c-838f-46e0-a439-00700619a558" (UID: "a365be8c-838f-46e0-a439-00700619a558"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:46:18.821077 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.820307 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-istio-token" pod "3e71bd07-c0e7-4013-ac2d-67206a09b3e2" (UID: "3e71bd07-c0e7-4013-ac2d-67206a09b3e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:46:18.821077 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.820558 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d22e1d9f-31a2-43a0-999c-084069d6a9eb-istio-data" pod "d22e1d9f-31a2-43a0-999c-084069d6a9eb" (UID: "d22e1d9f-31a2-43a0-999c-084069d6a9eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:46:18.821617 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.821570 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-istio-podinfo" pod "3e71bd07-c0e7-4013-ac2d-67206a09b3e2" (UID: "3e71bd07-c0e7-4013-ac2d-67206a09b3e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:46:18.821803 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.821649 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-istio-envoy" pod "3e71bd07-c0e7-4013-ac2d-67206a09b3e2" (UID: "3e71bd07-c0e7-4013-ac2d-67206a09b3e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:46:18.821803 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.821726 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a365be8c-838f-46e0-a439-00700619a558-istio-token" pod "a365be8c-838f-46e0-a439-00700619a558" (UID: "a365be8c-838f-46e0-a439-00700619a558"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:46:18.821803 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.821774 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d22e1d9f-31a2-43a0-999c-084069d6a9eb-istio-token" pod "d22e1d9f-31a2-43a0-999c-084069d6a9eb" (UID: "d22e1d9f-31a2-43a0-999c-084069d6a9eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:46:18.822165 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.822128 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a365be8c-838f-46e0-a439-00700619a558-istio-envoy" pod "a365be8c-838f-46e0-a439-00700619a558" (UID: "a365be8c-838f-46e0-a439-00700619a558"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:46:18.822693 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.822666 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d22e1d9f-31a2-43a0-999c-084069d6a9eb-istio-envoy" pod "d22e1d9f-31a2-43a0-999c-084069d6a9eb" (UID: "d22e1d9f-31a2-43a0-999c-084069d6a9eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:46:18.822992 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.822974 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d22e1d9f-31a2-43a0-999c-084069d6a9eb-kube-api-access-h4wk2" pod "d22e1d9f-31a2-43a0-999c-084069d6a9eb" (UID: "d22e1d9f-31a2-43a0-999c-084069d6a9eb"). InnerVolumeSpecName "kube-api-access-h4wk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:46:18.823118 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.823075 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-kube-api-access-b5hvv" pod "3e71bd07-c0e7-4013-ac2d-67206a09b3e2" (UID: "3e71bd07-c0e7-4013-ac2d-67206a09b3e2"). InnerVolumeSpecName "kube-api-access-b5hvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:46:18.823755 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.823732 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a365be8c-838f-46e0-a439-00700619a558-istio-podinfo" pod "a365be8c-838f-46e0-a439-00700619a558" (UID: "a365be8c-838f-46e0-a439-00700619a558"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:46:18.823835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.823753 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a365be8c-838f-46e0-a439-00700619a558-kube-api-access-w9f55" pod "a365be8c-838f-46e0-a439-00700619a558" (UID: "a365be8c-838f-46e0-a439-00700619a558"). InnerVolumeSpecName "kube-api-access-w9f55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:46:18.919433 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.919402 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-credential-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:18.919433 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.919428 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d22e1d9f-31a2-43a0-999c-084069d6a9eb-istio-token\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:18.919433 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.919438 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a365be8c-838f-46e0-a439-00700619a558-istio-token\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:18.919658 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.919447 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-workload-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:18.919658 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.919457 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a365be8c-838f-46e0-a439-00700619a558-istio-podinfo\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:18.919658 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.919466 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-workload-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:18.919658 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.919474 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a365be8c-838f-46e0-a439-00700619a558-credential-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:18.919658 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.919483 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d22e1d9f-31a2-43a0-999c-084069d6a9eb-istio-podinfo\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:18.919658 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.919491 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-istio-token\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:18.919658 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.919499 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-istiod-ca-cert\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:18.919658 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.919508 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-istio-podinfo\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:18.919658 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.919516 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5hvv\" (UniqueName: \"kubernetes.io/projected/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-kube-api-access-b5hvv\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:18.919658 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.919525 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9f55\" (UniqueName: \"kubernetes.io/projected/a365be8c-838f-46e0-a439-00700619a558-kube-api-access-w9f55\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:18.919658 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.919532 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d22e1d9f-31a2-43a0-999c-084069d6a9eb-istiod-ca-cert\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:18.919658 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.919540 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d22e1d9f-31a2-43a0-999c-084069d6a9eb-workload-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:18.919658 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.919549 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e71bd07-c0e7-4013-ac2d-67206a09b3e2-istio-envoy\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:18.919658 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.919558 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d22e1d9f-31a2-43a0-999c-084069d6a9eb-istio-envoy\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:18.919658 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.919566 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4wk2\" (UniqueName: \"kubernetes.io/projected/d22e1d9f-31a2-43a0-999c-084069d6a9eb-kube-api-access-h4wk2\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:18.919658 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.919574 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d22e1d9f-31a2-43a0-999c-084069d6a9eb-istio-data\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:18.919658 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.919587 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a365be8c-838f-46e0-a439-00700619a558-istio-envoy\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:18.919658 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.919600 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d22e1d9f-31a2-43a0-999c-084069d6a9eb-workload-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:18.919658 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:18.919616 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a365be8c-838f-46e0-a439-00700619a558-istio-data\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:19.337271 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:19.337231 4952 generic.go:356] "Generic (PLEG): container finished" podID="a365be8c-838f-46e0-a439-00700619a558" containerID="7f377bc9dec227a8d54c09e6ab0cc1bd3b94ffd7db12f3626d7ddf130245db28" exitCode=0 Oct 30 23:46:19.337457 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:19.337288 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" Oct 30 23:46:19.337457 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:19.337308 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" event={"ID":"a365be8c-838f-46e0-a439-00700619a558","Type":"ContainerDied","Data":"7f377bc9dec227a8d54c09e6ab0cc1bd3b94ffd7db12f3626d7ddf130245db28"} Oct 30 23:46:19.337457 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:19.337374 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2" event={"ID":"a365be8c-838f-46e0-a439-00700619a558","Type":"ContainerDied","Data":"edbe0239c8311685d1c540e4013a54517b6be42424c28df617b67da1d90a1582"} Oct 30 23:46:19.337457 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:19.337408 4952 scope.go:122] "RemoveContainer" containerID="7f377bc9dec227a8d54c09e6ab0cc1bd3b94ffd7db12f3626d7ddf130245db28" Oct 30 23:46:19.338996 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:19.338976 4952 generic.go:356] "Generic (PLEG): container finished" podID="d22e1d9f-31a2-43a0-999c-084069d6a9eb" containerID="18cc3d0fddc8a5f7c62694b0eabd1e51887acdbdcf4e8656a8d7eff92f6e6155" exitCode=0 Oct 30 23:46:19.339237 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:19.339077 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" Oct 30 23:46:19.339237 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:19.339092 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" event={"ID":"d22e1d9f-31a2-43a0-999c-084069d6a9eb","Type":"ContainerDied","Data":"18cc3d0fddc8a5f7c62694b0eabd1e51887acdbdcf4e8656a8d7eff92f6e6155"} Oct 30 23:46:19.339237 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:19.339153 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w" event={"ID":"d22e1d9f-31a2-43a0-999c-084069d6a9eb","Type":"ContainerDied","Data":"65fd4f9629c0a3f32e217ed18d02f6471ba069f45fd63cb7093af0109feb220d"} Oct 30 23:46:19.340977 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:19.340953 4952 generic.go:356] "Generic (PLEG): container finished" podID="3e71bd07-c0e7-4013-ac2d-67206a09b3e2" containerID="9f09a8cce7ab8093e99963d1f1f7688450135ec13a984d3057ad6023723b812a" exitCode=0 Oct 30 23:46:19.341123 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:19.341021 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" event={"ID":"3e71bd07-c0e7-4013-ac2d-67206a09b3e2","Type":"ContainerDied","Data":"9f09a8cce7ab8093e99963d1f1f7688450135ec13a984d3057ad6023723b812a"} Oct 30 23:46:19.341123 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:19.341103 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" event={"ID":"3e71bd07-c0e7-4013-ac2d-67206a09b3e2","Type":"ContainerDied","Data":"741695f371910b49a6331c867dfc785f6c1b09c2bd52beaee7bbdb17c91df78d"} Oct 30 23:46:19.341123 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:19.341115 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c" Oct 30 23:46:19.349823 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:19.349804 4952 scope.go:122] "RemoveContainer" containerID="7f377bc9dec227a8d54c09e6ab0cc1bd3b94ffd7db12f3626d7ddf130245db28" Oct 30 23:46:19.350187 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:46:19.350163 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f377bc9dec227a8d54c09e6ab0cc1bd3b94ffd7db12f3626d7ddf130245db28\": container with ID starting with 7f377bc9dec227a8d54c09e6ab0cc1bd3b94ffd7db12f3626d7ddf130245db28 not found: ID does not exist" containerID="7f377bc9dec227a8d54c09e6ab0cc1bd3b94ffd7db12f3626d7ddf130245db28" Oct 30 23:46:19.350257 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:19.350201 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f377bc9dec227a8d54c09e6ab0cc1bd3b94ffd7db12f3626d7ddf130245db28"} err="failed to get container status \"7f377bc9dec227a8d54c09e6ab0cc1bd3b94ffd7db12f3626d7ddf130245db28\": rpc error: code = NotFound desc = could not find container \"7f377bc9dec227a8d54c09e6ab0cc1bd3b94ffd7db12f3626d7ddf130245db28\": container with ID starting with 7f377bc9dec227a8d54c09e6ab0cc1bd3b94ffd7db12f3626d7ddf130245db28 not found: ID does not exist" Oct 30 23:46:19.350257 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:19.350228 4952 scope.go:122] "RemoveContainer" containerID="18cc3d0fddc8a5f7c62694b0eabd1e51887acdbdcf4e8656a8d7eff92f6e6155" Oct 30 23:46:19.361645 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:19.361628 4952 scope.go:122] "RemoveContainer" containerID="18cc3d0fddc8a5f7c62694b0eabd1e51887acdbdcf4e8656a8d7eff92f6e6155" Oct 30 23:46:19.361924 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:46:19.361897 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18cc3d0fddc8a5f7c62694b0eabd1e51887acdbdcf4e8656a8d7eff92f6e6155\": container with ID starting with 18cc3d0fddc8a5f7c62694b0eabd1e51887acdbdcf4e8656a8d7eff92f6e6155 not found: ID does not exist" containerID="18cc3d0fddc8a5f7c62694b0eabd1e51887acdbdcf4e8656a8d7eff92f6e6155" Oct 30 23:46:19.362003 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:19.361936 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18cc3d0fddc8a5f7c62694b0eabd1e51887acdbdcf4e8656a8d7eff92f6e6155"} err="failed to get container status \"18cc3d0fddc8a5f7c62694b0eabd1e51887acdbdcf4e8656a8d7eff92f6e6155\": rpc error: code = NotFound desc = could not find container \"18cc3d0fddc8a5f7c62694b0eabd1e51887acdbdcf4e8656a8d7eff92f6e6155\": container with ID starting with 18cc3d0fddc8a5f7c62694b0eabd1e51887acdbdcf4e8656a8d7eff92f6e6155 not found: ID does not exist" Oct 30 23:46:19.362003 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:19.361961 4952 scope.go:122] "RemoveContainer" containerID="9f09a8cce7ab8093e99963d1f1f7688450135ec13a984d3057ad6023723b812a" Oct 30 23:46:19.370850 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:19.370834 4952 scope.go:122] "RemoveContainer" containerID="9f09a8cce7ab8093e99963d1f1f7688450135ec13a984d3057ad6023723b812a" Oct 30 23:46:19.371173 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:46:19.371156 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f09a8cce7ab8093e99963d1f1f7688450135ec13a984d3057ad6023723b812a\": container with ID starting with 9f09a8cce7ab8093e99963d1f1f7688450135ec13a984d3057ad6023723b812a not found: ID does not exist" containerID="9f09a8cce7ab8093e99963d1f1f7688450135ec13a984d3057ad6023723b812a" Oct 30 23:46:19.371230 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:19.371180 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f09a8cce7ab8093e99963d1f1f7688450135ec13a984d3057ad6023723b812a"} err="failed to get container status \"9f09a8cce7ab8093e99963d1f1f7688450135ec13a984d3057ad6023723b812a\": rpc error: code = NotFound desc = could not find container \"9f09a8cce7ab8093e99963d1f1f7688450135ec13a984d3057ad6023723b812a\": container with ID starting with 9f09a8cce7ab8093e99963d1f1f7688450135ec13a984d3057ad6023723b812a not found: ID does not exist" Oct 30 23:46:19.501636 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:19.501548 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w"] Oct 30 23:46:19.508445 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:19.508424 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-q466w"] Oct 30 23:46:19.617254 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:19.617189 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2"] Oct 30 23:46:19.623449 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:19.623429 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-785d2"] Oct 30 23:46:19.731644 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:19.731618 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c"] Oct 30 23:46:19.738024 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:19.738002 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-f4p5c"] Oct 30 23:46:20.970257 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:20.970225 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e71bd07-c0e7-4013-ac2d-67206a09b3e2" path="/var/lib/kubelet/pods/3e71bd07-c0e7-4013-ac2d-67206a09b3e2/volumes" Oct 30 23:46:20.970698 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:20.970683 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a365be8c-838f-46e0-a439-00700619a558" path="/var/lib/kubelet/pods/a365be8c-838f-46e0-a439-00700619a558/volumes" Oct 30 23:46:20.971230 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:20.971210 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d22e1d9f-31a2-43a0-999c-084069d6a9eb" path="/var/lib/kubelet/pods/d22e1d9f-31a2-43a0-999c-084069d6a9eb/volumes" Oct 30 23:46:21.142320 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:46:21.142288 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:46:21.354418 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:21.354394 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft_91651ce2-4673-4ced-b26d-e94ff8829d11/extract/0.log" Oct 30 23:46:21.364884 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:21.364864 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/2.log" Oct 30 23:46:21.366521 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:21.366499 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft_91651ce2-4673-4ced-b26d-e94ff8829d11/extract/0.log" Oct 30 23:46:21.370952 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:21.370933 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 30 23:46:21.377381 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:21.377361 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/2.log" Oct 30 23:46:21.383358 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:21.383338 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 30 23:46:21.570240 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:21.570209 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:23.939728 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:23.939696 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:24.171499 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:24.171465 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:24.389350 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:24.389304 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:26.366246 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:26.366213 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:27.704311 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:27.704274 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:46:30.926866 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:30.926828 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:31.202777 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:31.202658 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:31.910383 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:31.910342 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:32.514168 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:32.514115 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:32.743848 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:32.743808 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:32.985254 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:32.985203 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:33.386401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:33.386366 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:36.289542 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.289511 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm"] Oct 30 23:46:36.290012 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290002 4952 cpu_manager.go:414] "containerMap: removing container" podUID="89959083-95eb-4626-b1b2-4b7ac9d76840" containerName="cleanup" Oct 30 23:46:36.290090 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290018 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89959083-95eb-4626-b1b2-4b7ac9d76840" containerName="cleanup" Oct 30 23:46:36.290090 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290050 4952 cpu_manager.go:414] "containerMap: removing container" podUID="032662ea-6f38-4092-8bda-d27d09daf88f" containerName="cleanup" Oct 30 23:46:36.290090 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290056 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="032662ea-6f38-4092-8bda-d27d09daf88f" containerName="cleanup" Oct 30 23:46:36.290090 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290066 4952 cpu_manager.go:414] "containerMap: removing container" podUID="54e5ee1a-c524-4f52-bc9c-870411e7f17e" containerName="cleanup" Oct 30 23:46:36.290090 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290071 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54e5ee1a-c524-4f52-bc9c-870411e7f17e" containerName="cleanup" Oct 30 23:46:36.290090 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290077 4952 cpu_manager.go:414] "containerMap: removing container" podUID="54e5ee1a-c524-4f52-bc9c-870411e7f17e" containerName="cleanup" Oct 30 23:46:36.290090 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290081 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54e5ee1a-c524-4f52-bc9c-870411e7f17e" containerName="cleanup" Oct 30 23:46:36.290090 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290091 4952 cpu_manager.go:414] "containerMap: removing container" podUID="032662ea-6f38-4092-8bda-d27d09daf88f" containerName="cleanup" Oct 30 23:46:36.290338 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290096 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="032662ea-6f38-4092-8bda-d27d09daf88f" containerName="cleanup" Oct 30 23:46:36.290338 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290103 4952 cpu_manager.go:414] "containerMap: removing container" podUID="3b443f22-9bac-4a46-9b4a-8982e0006fc4" containerName="payload-pre-processing" Oct 30 23:46:36.290338 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290108 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b443f22-9bac-4a46-9b4a-8982e0006fc4" containerName="payload-pre-processing" Oct 30 23:46:36.290338 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290114 4952 cpu_manager.go:414] "containerMap: removing container" podUID="5cd22a14-b583-4991-8ced-45c5b15919ba" containerName="cleanup" Oct 30 23:46:36.290338 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290118 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cd22a14-b583-4991-8ced-45c5b15919ba" containerName="cleanup" Oct 30 23:46:36.290338 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290125 4952 cpu_manager.go:414] "containerMap: removing container" podUID="44adfd6a-2626-4378-bca1-d46ee1670b07" containerName="main" Oct 30 23:46:36.290338 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290131 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44adfd6a-2626-4378-bca1-d46ee1670b07" containerName="main" Oct 30 23:46:36.290338 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290138 4952 cpu_manager.go:414] "containerMap: removing container" podUID="54e5ee1a-c524-4f52-bc9c-870411e7f17e" containerName="cleanup" Oct 30 23:46:36.290338 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290144 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54e5ee1a-c524-4f52-bc9c-870411e7f17e" containerName="cleanup" Oct 30 23:46:36.290338 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290152 4952 cpu_manager.go:414] "containerMap: removing container" podUID="ec9e7fff-1e25-4f16-ab95-f1a1d65448da" containerName="istio-proxy" Oct 30 23:46:36.290338 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290157 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec9e7fff-1e25-4f16-ab95-f1a1d65448da" containerName="istio-proxy" Oct 30 23:46:36.290338 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290166 4952 cpu_manager.go:414] "containerMap: removing container" podUID="3e71bd07-c0e7-4013-ac2d-67206a09b3e2" containerName="istio-proxy" Oct 30 23:46:36.290338 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290170 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e71bd07-c0e7-4013-ac2d-67206a09b3e2" containerName="istio-proxy" Oct 30 23:46:36.290338 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290183 4952 cpu_manager.go:414] "containerMap: removing container" podUID="4f85606f-6cb6-442e-a5e1-3a57fb732f93" containerName="maas-api" Oct 30 23:46:36.290338 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290187 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f85606f-6cb6-442e-a5e1-3a57fb732f93" containerName="maas-api" Oct 30 23:46:36.290338 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290199 4952 cpu_manager.go:414] "containerMap: removing container" podUID="a365be8c-838f-46e0-a439-00700619a558" containerName="istio-proxy" Oct 30 23:46:36.290338 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290204 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a365be8c-838f-46e0-a439-00700619a558" containerName="istio-proxy" Oct 30 23:46:36.290338 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290213 4952 cpu_manager.go:414] "containerMap: removing container" podUID="89959083-95eb-4626-b1b2-4b7ac9d76840" containerName="cleanup" Oct 30 23:46:36.290338 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290217 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89959083-95eb-4626-b1b2-4b7ac9d76840" containerName="cleanup" Oct 30 23:46:36.290338 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290222 4952 cpu_manager.go:414] "containerMap: removing container" podUID="032662ea-6f38-4092-8bda-d27d09daf88f" containerName="cleanup" Oct 30 23:46:36.290338 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290227 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="032662ea-6f38-4092-8bda-d27d09daf88f" containerName="cleanup" Oct 30 23:46:36.290338 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290235 4952 cpu_manager.go:414] "containerMap: removing container" podUID="5cd22a14-b583-4991-8ced-45c5b15919ba" containerName="cleanup" Oct 30 23:46:36.290338 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290239 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cd22a14-b583-4991-8ced-45c5b15919ba" containerName="cleanup" Oct 30 23:46:36.290338 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290248 4952 cpu_manager.go:414] "containerMap: removing container" podUID="d22e1d9f-31a2-43a0-999c-084069d6a9eb" containerName="istio-proxy" Oct 30 23:46:36.290338 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290252 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d22e1d9f-31a2-43a0-999c-084069d6a9eb" containerName="istio-proxy" Oct 30 23:46:36.290338 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290261 4952 cpu_manager.go:414] "containerMap: removing container" podUID="43e39353-ae85-4d76-9900-7e7927409098" containerName="main" Oct 30 23:46:36.290338 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290265 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43e39353-ae85-4d76-9900-7e7927409098" containerName="main" Oct 30 23:46:36.290338 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290273 4952 cpu_manager.go:414] "containerMap: removing container" podUID="28913940-90f7-4d80-89b7-f5eece975d02" containerName="main" Oct 30 23:46:36.290338 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290277 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28913940-90f7-4d80-89b7-f5eece975d02" containerName="main" Oct 30 23:46:36.290338 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290283 4952 cpu_manager.go:414] "containerMap: removing container" podUID="f90492f3-9a0a-447d-8a29-6a91e026a5b1" containerName="maas-api" Oct 30 23:46:36.290338 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290287 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f90492f3-9a0a-447d-8a29-6a91e026a5b1" containerName="maas-api" Oct 30 23:46:36.291340 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290377 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="89959083-95eb-4626-b1b2-4b7ac9d76840" containerName="cleanup" Oct 30 23:46:36.291340 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290385 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f85606f-6cb6-442e-a5e1-3a57fb732f93" containerName="maas-api" Oct 30 23:46:36.291340 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290392 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cd22a14-b583-4991-8ced-45c5b15919ba" containerName="cleanup" Oct 30 23:46:36.291340 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290399 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="a365be8c-838f-46e0-a439-00700619a558" containerName="istio-proxy" Oct 30 23:46:36.291340 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290404 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="89959083-95eb-4626-b1b2-4b7ac9d76840" containerName="cleanup" Oct 30 23:46:36.291340 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290410 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="032662ea-6f38-4092-8bda-d27d09daf88f" containerName="cleanup" Oct 30 23:46:36.291340 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290416 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cd22a14-b583-4991-8ced-45c5b15919ba" containerName="cleanup" Oct 30 23:46:36.291340 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290421 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="d22e1d9f-31a2-43a0-999c-084069d6a9eb" containerName="istio-proxy" Oct 30 23:46:36.291340 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290426 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="43e39353-ae85-4d76-9900-7e7927409098" containerName="main" Oct 30 23:46:36.291340 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290431 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="28913940-90f7-4d80-89b7-f5eece975d02" containerName="main" Oct 30 23:46:36.291340 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290435 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="f90492f3-9a0a-447d-8a29-6a91e026a5b1" containerName="maas-api" Oct 30 23:46:36.291340 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290442 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="032662ea-6f38-4092-8bda-d27d09daf88f" containerName="cleanup" Oct 30 23:46:36.291340 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290448 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="54e5ee1a-c524-4f52-bc9c-870411e7f17e" containerName="cleanup" Oct 30 23:46:36.291340 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290453 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="54e5ee1a-c524-4f52-bc9c-870411e7f17e" containerName="cleanup" Oct 30 23:46:36.291340 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290458 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="032662ea-6f38-4092-8bda-d27d09daf88f" containerName="cleanup" Oct 30 23:46:36.291340 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290463 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b443f22-9bac-4a46-9b4a-8982e0006fc4" containerName="payload-pre-processing" Oct 30 23:46:36.291340 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290469 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="44adfd6a-2626-4378-bca1-d46ee1670b07" containerName="main" Oct 30 23:46:36.291340 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290473 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="54e5ee1a-c524-4f52-bc9c-870411e7f17e" containerName="cleanup" Oct 30 23:46:36.291340 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290477 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec9e7fff-1e25-4f16-ab95-f1a1d65448da" containerName="istio-proxy" Oct 30 23:46:36.291340 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.290483 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e71bd07-c0e7-4013-ac2d-67206a09b3e2" containerName="istio-proxy" Oct 30 23:46:36.296414 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.296387 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" Oct 30 23:46:36.317290 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.317260 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm"] Oct 30 23:46:36.329641 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.329615 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t6s9\" (UniqueName: \"kubernetes.io/projected/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-kube-api-access-8t6s9\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" Oct 30 23:46:36.329769 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.329720 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-istio-podinfo\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" Oct 30 23:46:36.329769 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.329742 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-workload-socket\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" Oct 30 23:46:36.329843 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.329820 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-istiod-ca-cert\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" Oct 30 23:46:36.329931 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.329901 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-istio-token\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" Oct 30 23:46:36.330006 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.329994 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-workload-certs\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" Oct 30 23:46:36.330119 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.330096 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-credential-socket\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" Oct 30 23:46:36.330160 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.330142 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-istio-data\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" Oct 30 23:46:36.330199 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.330175 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-istio-envoy\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" Oct 30 23:46:36.405749 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.405714 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-71a68a-openshift-default-dockercfg-hdm8z\"" Oct 30 23:46:36.431579 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.431546 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-istio-podinfo\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" Oct 30 23:46:36.431743 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.431589 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-workload-socket\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" Oct 30 23:46:36.431743 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.431658 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-istiod-ca-cert\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" Oct 30 23:46:36.431743 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.431697 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-istio-token\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" Oct 30 23:46:36.431922 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.431750 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-workload-certs\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" Oct 30 23:46:36.431922 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.431803 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-credential-socket\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" Oct 30 23:46:36.431922 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.431826 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-istio-data\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" Oct 30 23:46:36.431922 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.431852 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-istio-envoy\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" Oct 30 23:46:36.431922 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.431883 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t6s9\" (UniqueName: \"kubernetes.io/projected/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-kube-api-access-8t6s9\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" Oct 30 23:46:36.432350 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.432298 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-workload-socket\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" Oct 30 23:46:36.432575 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.432456 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-workload-certs\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" Oct 30 23:46:36.432787 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.432585 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-istiod-ca-cert\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" Oct 30 23:46:36.433480 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.433457 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-credential-socket\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" Oct 30 23:46:36.433732 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.433712 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-istio-data\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" Oct 30 23:46:36.435122 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.435099 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-istio-podinfo\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" Oct 30 23:46:36.435410 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.435385 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-istio-envoy\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" Oct 30 23:46:36.493426 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.493394 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-istio-token\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" Oct 30 23:46:36.493663 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.493638 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t6s9\" (UniqueName: \"kubernetes.io/projected/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-kube-api-access-8t6s9\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" Oct 30 23:46:36.615705 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.615681 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" Oct 30 23:46:36.788945 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:36.788880 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:37.029725 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:37.029635 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:37.046989 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:37.046965 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm"] Oct 30 23:46:37.048376 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:46:37.048340 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d92b1e_2565_4881_ae9e_c605e91aa1b8.slice/crio-ec18df05ca663040f4fe74b9c8dcf4aeb1c5fb18a7eb3fc2611cfc191e0e1b61 WatchSource:0}: Error finding container ec18df05ca663040f4fe74b9c8dcf4aeb1c5fb18a7eb3fc2611cfc191e0e1b61: Status 404 returned error can't find the container with id ec18df05ca663040f4fe74b9c8dcf4aeb1c5fb18a7eb3fc2611cfc191e0e1b61 Oct 30 23:46:37.050725 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:37.050690 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:46:37.050795 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:37.050759 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:46:37.050795 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:37.050791 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:46:37.175859 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:37.175831 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8"] Oct 30 23:46:37.176087 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:37.176060 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8" podUID="09eac6de-60af-424f-92e6-b230e63a3712" containerName="maas-api" containerID="cri-o://c8529780cf2210292279647210b8e7c0d0271e2f81f5480a36cf398d6b6d3056" gracePeriod=30 Oct 30 23:46:37.389105 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:37.389006 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:37.500227 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:37.500196 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8" Oct 30 23:46:37.516027 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:37.515994 4952 generic.go:356] "Generic (PLEG): container finished" podID="09eac6de-60af-424f-92e6-b230e63a3712" containerID="c8529780cf2210292279647210b8e7c0d0271e2f81f5480a36cf398d6b6d3056" exitCode=0 Oct 30 23:46:37.516027 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:37.516053 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8" Oct 30 23:46:37.516268 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:37.516092 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8" event={"ID":"09eac6de-60af-424f-92e6-b230e63a3712","Type":"ContainerDied","Data":"c8529780cf2210292279647210b8e7c0d0271e2f81f5480a36cf398d6b6d3056"} Oct 30 23:46:37.516268 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:37.516153 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8" event={"ID":"09eac6de-60af-424f-92e6-b230e63a3712","Type":"ContainerDied","Data":"0a205cb7a72f8ba40abba290673bf2c7052a76f94964cd81c3641aaf0c823c0b"} Oct 30 23:46:37.516268 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:37.516193 4952 scope.go:122] "RemoveContainer" containerID="c8529780cf2210292279647210b8e7c0d0271e2f81f5480a36cf398d6b6d3056" Oct 30 23:46:37.518258 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:37.518231 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" event={"ID":"c1d92b1e-2565-4881-ae9e-c605e91aa1b8","Type":"ContainerStarted","Data":"b50f76e5c0db692dc142780d07e8ccc16be9e0d0dc553f697d3f42981e1d8e54"} Oct 30 23:46:37.518370 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:37.518288 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" event={"ID":"c1d92b1e-2565-4881-ae9e-c605e91aa1b8","Type":"ContainerStarted","Data":"ec18df05ca663040f4fe74b9c8dcf4aeb1c5fb18a7eb3fc2611cfc191e0e1b61"} Oct 30 23:46:37.537230 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:37.536796 4952 scope.go:122] "RemoveContainer" containerID="c8529780cf2210292279647210b8e7c0d0271e2f81f5480a36cf398d6b6d3056" Oct 30 23:46:37.538726 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:46:37.538695 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8529780cf2210292279647210b8e7c0d0271e2f81f5480a36cf398d6b6d3056\": container with ID starting with c8529780cf2210292279647210b8e7c0d0271e2f81f5480a36cf398d6b6d3056 not found: ID does not exist" containerID="c8529780cf2210292279647210b8e7c0d0271e2f81f5480a36cf398d6b6d3056" Oct 30 23:46:37.538857 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:37.538743 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8529780cf2210292279647210b8e7c0d0271e2f81f5480a36cf398d6b6d3056"} err="failed to get container status \"c8529780cf2210292279647210b8e7c0d0271e2f81f5480a36cf398d6b6d3056\": rpc error: code = NotFound desc = could not find container \"c8529780cf2210292279647210b8e7c0d0271e2f81f5480a36cf398d6b6d3056\": container with ID starting with c8529780cf2210292279647210b8e7c0d0271e2f81f5480a36cf398d6b6d3056 not found: ID does not exist" Oct 30 23:46:37.543206 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:37.543181 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09eac6de-60af-424f-92e6-b230e63a3712-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09eac6de-60af-424f-92e6-b230e63a3712-openshift-service-ca\") pod \"09eac6de-60af-424f-92e6-b230e63a3712\" (UID: \"09eac6de-60af-424f-92e6-b230e63a3712\") " Oct 30 23:46:37.543337 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:37.543217 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09eac6de-60af-424f-92e6-b230e63a3712-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09eac6de-60af-424f-92e6-b230e63a3712-maas-api-tls\") pod \"09eac6de-60af-424f-92e6-b230e63a3712\" (UID: \"09eac6de-60af-424f-92e6-b230e63a3712\") " Oct 30 23:46:37.543337 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:37.543306 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09eac6de-60af-424f-92e6-b230e63a3712-kube-api-access-k5x94\" (UniqueName: \"kubernetes.io/projected/09eac6de-60af-424f-92e6-b230e63a3712-kube-api-access-k5x94\") pod \"09eac6de-60af-424f-92e6-b230e63a3712\" (UID: \"09eac6de-60af-424f-92e6-b230e63a3712\") " Oct 30 23:46:37.543448 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:37.543374 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09eac6de-60af-424f-92e6-b230e63a3712-metrics-certs\" (UniqueName: \"kubernetes.io/secret/09eac6de-60af-424f-92e6-b230e63a3712-metrics-certs\") pod \"09eac6de-60af-424f-92e6-b230e63a3712\" (UID: \"09eac6de-60af-424f-92e6-b230e63a3712\") " Oct 30 23:46:37.545101 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:37.545015 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09eac6de-60af-424f-92e6-b230e63a3712-openshift-service-ca" pod "09eac6de-60af-424f-92e6-b230e63a3712" (UID: "09eac6de-60af-424f-92e6-b230e63a3712"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:46:37.551769 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:37.547162 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09eac6de-60af-424f-92e6-b230e63a3712-maas-api-tls" pod "09eac6de-60af-424f-92e6-b230e63a3712" (UID: "09eac6de-60af-424f-92e6-b230e63a3712"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:46:37.558282 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:37.558226 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09eac6de-60af-424f-92e6-b230e63a3712-kube-api-access-k5x94" pod "09eac6de-60af-424f-92e6-b230e63a3712" (UID: "09eac6de-60af-424f-92e6-b230e63a3712"). InnerVolumeSpecName "kube-api-access-k5x94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:46:37.576144 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:37.575016 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09eac6de-60af-424f-92e6-b230e63a3712-metrics-certs" pod "09eac6de-60af-424f-92e6-b230e63a3712" (UID: "09eac6de-60af-424f-92e6-b230e63a3712"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:46:37.615816 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:37.615774 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" Oct 30 23:46:37.645026 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:37.644993 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09eac6de-60af-424f-92e6-b230e63a3712-openshift-service-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:37.645026 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:37.645023 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09eac6de-60af-424f-92e6-b230e63a3712-maas-api-tls\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:37.645026 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:37.645054 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5x94\" (UniqueName: \"kubernetes.io/projected/09eac6de-60af-424f-92e6-b230e63a3712-kube-api-access-k5x94\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:37.645357 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:37.645069 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09eac6de-60af-424f-92e6-b230e63a3712-metrics-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:46:37.749061 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:37.747697 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" podStartSLOduration=1.747677612 podStartE2EDuration="1.747677612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:46:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:37.738372759 +0000 UTC m=+2416.882202977" watchObservedRunningTime="2026-10-30 23:46:37.747677612 +0000 UTC m=+2416.891507818" Oct 30 23:46:37.938255 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:37.938215 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8"] Oct 30 23:46:37.942350 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:37.942317 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-26ce6b-d4bd9858b-x48m8"] Oct 30 23:46:38.618184 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:38.618144 4952 patch_prober.go:29] interesting pod/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 23:46:38.618585 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:38.618199 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" podUID="c1d92b1e-2565-4881-ae9e-c605e91aa1b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 23:46:38.987112 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:38.976121 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09eac6de-60af-424f-92e6-b230e63a3712" path="/var/lib/kubelet/pods/09eac6de-60af-424f-92e6-b230e63a3712/volumes" Oct 30 23:46:39.225598 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:39.225567 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:39.616531 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:39.616497 4952 patch_prober.go:29] interesting pod/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 23:46:39.616721 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:39.616546 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" podUID="c1d92b1e-2565-4881-ae9e-c605e91aa1b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 23:46:40.616268 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:40.616233 4952 patch_prober.go:29] interesting pod/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 23:46:40.616658 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:40.616283 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" podUID="c1d92b1e-2565-4881-ae9e-c605e91aa1b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 23:46:41.247822 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:41.247785 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:41.616937 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:41.616900 4952 patch_prober.go:29] interesting pod/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 23:46:41.617384 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:41.616979 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" podUID="c1d92b1e-2565-4881-ae9e-c605e91aa1b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 23:46:41.686061 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:41.682321 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:42.616409 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:42.616379 4952 patch_prober.go:29] interesting pod/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 23:46:42.616586 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:42.616431 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" podUID="c1d92b1e-2565-4881-ae9e-c605e91aa1b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 23:46:43.622257 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:43.622226 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" Oct 30 23:46:43.623226 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:43.623082 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" Oct 30 23:46:43.625992 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:43.625969 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" Oct 30 23:46:44.171963 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:44.171902 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:44.409517 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:44.409483 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:44.653607 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:44.653557 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:46.617068 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:46.617008 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:46.851791 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:46.851751 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:47.384462 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:47.384374 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:47.612878 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:47.612842 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:47.926231 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:47.926157 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:48.406846 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:48.406809 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:48.634685 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:48.634652 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:48.868349 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:48.868319 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:49.109203 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:49.109167 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:49.784954 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:49.784918 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:50.007656 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.007605 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:50.444275 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.444242 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc"] Oct 30 23:46:50.447938 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.447909 4952 cpu_manager.go:414] "containerMap: removing container" podUID="09eac6de-60af-424f-92e6-b230e63a3712" containerName="maas-api" Oct 30 23:46:50.448263 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.448239 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09eac6de-60af-424f-92e6-b230e63a3712" containerName="maas-api" Oct 30 23:46:50.448404 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.448308 4952 cpu_manager.go:414] "containerMap: removing container" podUID="5cd22a14-b583-4991-8ced-45c5b15919ba" containerName="cleanup" Oct 30 23:46:50.448404 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.448317 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cd22a14-b583-4991-8ced-45c5b15919ba" containerName="cleanup" Oct 30 23:46:50.448404 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.448363 4952 cpu_manager.go:414] "containerMap: removing container" podUID="89959083-95eb-4626-b1b2-4b7ac9d76840" containerName="cleanup" Oct 30 23:46:50.448404 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.448372 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89959083-95eb-4626-b1b2-4b7ac9d76840" containerName="cleanup" Oct 30 23:46:50.448873 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.448509 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cd22a14-b583-4991-8ced-45c5b15919ba" containerName="cleanup" Oct 30 23:46:50.448873 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.448524 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="09eac6de-60af-424f-92e6-b230e63a3712" containerName="maas-api" Oct 30 23:46:50.448873 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.448546 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="89959083-95eb-4626-b1b2-4b7ac9d76840" containerName="cleanup" Oct 30 23:46:50.452449 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.452432 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" Oct 30 23:46:50.506573 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.506539 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc"] Oct 30 23:46:50.548939 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.548910 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/414e5e6c-0f02-40f5-806f-0ad048215d6a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vgtcc\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" Oct 30 23:46:50.548939 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.548948 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/414e5e6c-0f02-40f5-806f-0ad048215d6a-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vgtcc\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" Oct 30 23:46:50.549205 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.548974 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/414e5e6c-0f02-40f5-806f-0ad048215d6a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vgtcc\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" Oct 30 23:46:50.549205 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.549136 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/414e5e6c-0f02-40f5-806f-0ad048215d6a-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vgtcc\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" Oct 30 23:46:50.549205 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.549185 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/414e5e6c-0f02-40f5-806f-0ad048215d6a-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vgtcc\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" Oct 30 23:46:50.549380 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.549325 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn8wf\" (UniqueName: \"kubernetes.io/projected/414e5e6c-0f02-40f5-806f-0ad048215d6a-kube-api-access-rn8wf\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vgtcc\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" Oct 30 23:46:50.549442 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.549426 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/414e5e6c-0f02-40f5-806f-0ad048215d6a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vgtcc\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" Oct 30 23:46:50.549534 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.549519 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/414e5e6c-0f02-40f5-806f-0ad048215d6a-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vgtcc\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" Oct 30 23:46:50.549598 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.549579 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/414e5e6c-0f02-40f5-806f-0ad048215d6a-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vgtcc\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" Oct 30 23:46:50.651339 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.651288 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn8wf\" (UniqueName: \"kubernetes.io/projected/414e5e6c-0f02-40f5-806f-0ad048215d6a-kube-api-access-rn8wf\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vgtcc\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" Oct 30 23:46:50.651560 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.651386 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/414e5e6c-0f02-40f5-806f-0ad048215d6a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vgtcc\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" Oct 30 23:46:50.651560 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.651467 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/414e5e6c-0f02-40f5-806f-0ad048215d6a-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vgtcc\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" Oct 30 23:46:50.651560 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.651507 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/414e5e6c-0f02-40f5-806f-0ad048215d6a-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vgtcc\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" Oct 30 23:46:50.651742 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.651569 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/414e5e6c-0f02-40f5-806f-0ad048215d6a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vgtcc\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" Oct 30 23:46:50.651742 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.651603 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/414e5e6c-0f02-40f5-806f-0ad048215d6a-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vgtcc\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" Oct 30 23:46:50.651917 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.651894 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/414e5e6c-0f02-40f5-806f-0ad048215d6a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vgtcc\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" Oct 30 23:46:50.651989 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.651916 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/414e5e6c-0f02-40f5-806f-0ad048215d6a-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vgtcc\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" Oct 30 23:46:50.651989 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.651942 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/414e5e6c-0f02-40f5-806f-0ad048215d6a-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vgtcc\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" Oct 30 23:46:50.651989 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.651978 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/414e5e6c-0f02-40f5-806f-0ad048215d6a-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vgtcc\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" Oct 30 23:46:50.652206 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.651997 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/414e5e6c-0f02-40f5-806f-0ad048215d6a-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vgtcc\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" Oct 30 23:46:50.652283 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.652261 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/414e5e6c-0f02-40f5-806f-0ad048215d6a-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vgtcc\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" Oct 30 23:46:50.652391 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.652369 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/414e5e6c-0f02-40f5-806f-0ad048215d6a-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vgtcc\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" Oct 30 23:46:50.652446 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.652426 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/414e5e6c-0f02-40f5-806f-0ad048215d6a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vgtcc\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" Oct 30 23:46:50.653934 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.653912 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/414e5e6c-0f02-40f5-806f-0ad048215d6a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vgtcc\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" Oct 30 23:46:50.654688 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.654667 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/414e5e6c-0f02-40f5-806f-0ad048215d6a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vgtcc\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" Oct 30 23:46:50.708529 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.708453 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/414e5e6c-0f02-40f5-806f-0ad048215d6a-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vgtcc\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" Oct 30 23:46:50.708658 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.708607 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn8wf\" (UniqueName: \"kubernetes.io/projected/414e5e6c-0f02-40f5-806f-0ad048215d6a-kube-api-access-rn8wf\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vgtcc\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" Oct 30 23:46:50.769117 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:50.769071 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" Oct 30 23:46:51.143103 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:46:51.143066 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:46:51.212895 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:51.212857 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc"] Oct 30 23:46:51.215078 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:46:51.215022 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod414e5e6c_0f02_40f5_806f_0ad048215d6a.slice/crio-d86e976cd190c2fe4e091ba72006fdf059ea9126e7c5e3b76ea4b1bd3d2541c6 WatchSource:0}: Error finding container d86e976cd190c2fe4e091ba72006fdf059ea9126e7c5e3b76ea4b1bd3d2541c6: Status 404 returned error can't find the container with id d86e976cd190c2fe4e091ba72006fdf059ea9126e7c5e3b76ea4b1bd3d2541c6 Oct 30 23:46:51.217617 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:51.217582 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:46:51.217697 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:51.217650 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:46:51.217697 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:51.217678 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 23:46:51.599061 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:51.591442 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:51.642098 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:51.638263 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" event={"ID":"414e5e6c-0f02-40f5-806f-0ad048215d6a","Type":"ContainerStarted","Data":"4b24d984ef4a14d826ad729bd5e747c1e60a60997ed9f183eed3714200becf3d"} Oct 30 23:46:51.642098 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:51.638328 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" event={"ID":"414e5e6c-0f02-40f5-806f-0ad048215d6a","Type":"ContainerStarted","Data":"d86e976cd190c2fe4e091ba72006fdf059ea9126e7c5e3b76ea4b1bd3d2541c6"} Oct 30 23:46:51.770660 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:51.770620 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" Oct 30 23:46:51.772886 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:51.772367 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" podStartSLOduration=1.772347619 podStartE2EDuration="1.772347619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:51.757068589 +0000 UTC m=+2430.900898797" watchObservedRunningTime="2026-10-30 23:46:51.772347619 +0000 UTC m=+2430.916177827" Oct 30 23:46:51.831094 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:51.830791 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:52.790064 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:52.789615 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-vgtcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 23:46:52.790064 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:52.789677 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" podUID="414e5e6c-0f02-40f5-806f-0ad048215d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 23:46:53.118230 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:53.118198 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:53.401257 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:53.401176 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:53.782117 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:53.775417 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-vgtcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 23:46:53.782117 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:53.775474 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" podUID="414e5e6c-0f02-40f5-806f-0ad048215d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 23:46:53.823589 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:53.823554 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:54.070117 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:54.069748 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:54.772075 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:54.770377 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-vgtcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 23:46:54.772075 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:54.770470 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" podUID="414e5e6c-0f02-40f5-806f-0ad048215d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 23:46:55.263467 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:55.263435 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:55.787438 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:55.787400 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-vgtcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 23:46:55.787742 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:55.787713 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" podUID="414e5e6c-0f02-40f5-806f-0ad048215d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 23:46:56.786065 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:56.772381 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-vgtcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 23:46:56.786065 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:56.772445 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" podUID="414e5e6c-0f02-40f5-806f-0ad048215d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 23:46:57.770290 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:57.770253 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-vgtcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 23:46:57.770465 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:57.770312 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" podUID="414e5e6c-0f02-40f5-806f-0ad048215d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 23:46:58.749347 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:58.749311 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:58.774055 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:58.774004 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-vgtcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 23:46:58.774374 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:58.774326 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" podUID="414e5e6c-0f02-40f5-806f-0ad048215d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 23:46:58.978068 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:58.977993 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:46:59.911031 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:59.910993 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" Oct 30 23:46:59.911557 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:59.911332 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" Oct 30 23:46:59.912388 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:46:59.912363 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" Oct 30 23:47:00.635327 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:00.635291 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:01.293639 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:01.293599 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:01.522046 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:01.521998 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:05.888502 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:05.888464 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:11.004313 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:11.004281 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:15.875490 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:15.875455 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:20.770451 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:20.770419 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:21.143507 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:47:21.143473 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:47:24.515593 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:24.515566 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:24.550992 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:24.550959 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:47:25.562161 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:25.562116 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:30.299764 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:30.299676 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:32.937365 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:32.937325 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:35.223383 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:35.223350 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:35.454703 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:35.454670 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:35.672426 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:35.672388 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:37.819867 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:37.819832 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt"] Oct 30 23:47:38.069007 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:38.068971 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:40.547281 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:40.547246 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:44.088966 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:44.088929 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3c9613-75d9965b8c-g924m"] Oct 30 23:47:44.453150 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:44.437392 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:44.689829 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:44.689777 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:45.656247 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:45.656217 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k"] Oct 30 23:47:45.660107 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:45.660086 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k" Oct 30 23:47:45.764351 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:45.764320 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-15b137-serving-cert\"" Oct 30 23:47:45.768396 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:45.768372 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k"] Oct 30 23:47:45.827517 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:45.827468 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klw7c\" (UniqueName: \"kubernetes.io/projected/d534649f-0944-4efd-b936-069af23b30ba-kube-api-access-klw7c\") pod \"maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k\" (UID: \"d534649f-0944-4efd-b936-069af23b30ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k" Oct 30 23:47:45.828502 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:45.828461 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d534649f-0944-4efd-b936-069af23b30ba-metrics-certs\") pod \"maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k\" (UID: \"d534649f-0944-4efd-b936-069af23b30ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k" Oct 30 23:47:45.828775 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:45.828758 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d534649f-0944-4efd-b936-069af23b30ba-maas-api-tls\") pod \"maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k\" (UID: \"d534649f-0944-4efd-b936-069af23b30ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k" Oct 30 23:47:45.828960 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:45.828929 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d534649f-0944-4efd-b936-069af23b30ba-openshift-service-ca\") pod \"maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k\" (UID: \"d534649f-0944-4efd-b936-069af23b30ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k" Oct 30 23:47:45.930059 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:45.929942 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d534649f-0944-4efd-b936-069af23b30ba-metrics-certs\") pod \"maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k\" (UID: \"d534649f-0944-4efd-b936-069af23b30ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k" Oct 30 23:47:45.930426 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:45.930397 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d534649f-0944-4efd-b936-069af23b30ba-maas-api-tls\") pod \"maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k\" (UID: \"d534649f-0944-4efd-b936-069af23b30ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k" Oct 30 23:47:45.935344 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:45.935314 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d534649f-0944-4efd-b936-069af23b30ba-openshift-service-ca\") pod \"maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k\" (UID: \"d534649f-0944-4efd-b936-069af23b30ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k" Oct 30 23:47:45.935552 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:45.935525 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klw7c\" (UniqueName: \"kubernetes.io/projected/d534649f-0944-4efd-b936-069af23b30ba-kube-api-access-klw7c\") pod \"maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k\" (UID: \"d534649f-0944-4efd-b936-069af23b30ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k" Oct 30 23:47:45.936021 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:47:45.930637 4952 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-15b137-serving-cert: secret "maas-api-e2e-inf-b-15b137-serving-cert" not found Oct 30 23:47:45.936115 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:45.936079 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d534649f-0944-4efd-b936-069af23b30ba-openshift-service-ca\") pod \"maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k\" (UID: \"d534649f-0944-4efd-b936-069af23b30ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k" Oct 30 23:47:45.936279 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:47:45.936163 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d534649f-0944-4efd-b936-069af23b30ba-maas-api-tls podName:d534649f-0944-4efd-b936-069af23b30ba nodeName:}" failed. No retries permitted until 2026-10-30 23:47:46.436131096 +0000 UTC m=+2485.579961351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d534649f-0944-4efd-b936-069af23b30ba-maas-api-tls") pod "maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k" (UID: "d534649f-0944-4efd-b936-069af23b30ba") : secret "maas-api-e2e-inf-b-15b137-serving-cert" not found Oct 30 23:47:45.938977 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:45.938489 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d534649f-0944-4efd-b936-069af23b30ba-metrics-certs\") pod \"maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k\" (UID: \"d534649f-0944-4efd-b936-069af23b30ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k" Oct 30 23:47:45.949825 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:45.949787 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:45.995794 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:45.995761 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klw7c\" (UniqueName: \"kubernetes.io/projected/d534649f-0944-4efd-b936-069af23b30ba-kube-api-access-klw7c\") pod \"maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k\" (UID: \"d534649f-0944-4efd-b936-069af23b30ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k" Oct 30 23:47:46.180931 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:46.180841 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:46.440548 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:46.440466 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d534649f-0944-4efd-b936-069af23b30ba-maas-api-tls\") pod \"maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k\" (UID: \"d534649f-0944-4efd-b936-069af23b30ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k" Oct 30 23:47:46.443123 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:46.443101 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d534649f-0944-4efd-b936-069af23b30ba-maas-api-tls\") pod \"maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k\" (UID: \"d534649f-0944-4efd-b936-069af23b30ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k" Oct 30 23:47:46.590453 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:46.590414 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k" Oct 30 23:47:46.645494 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:46.645459 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:46.974699 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:46.974671 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k"] Oct 30 23:47:46.976010 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:47:46.975984 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd534649f_0944_4efd_b936_069af23b30ba.slice/crio-fae9a2a4b25c4ff64351ed6433d2c62cc856f9a9595c8413f7e2948f06d65ec2 WatchSource:0}: Error finding container fae9a2a4b25c4ff64351ed6433d2c62cc856f9a9595c8413f7e2948f06d65ec2: Status 404 returned error can't find the container with id fae9a2a4b25c4ff64351ed6433d2c62cc856f9a9595c8413f7e2948f06d65ec2 Oct 30 23:47:46.977394 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:46.977374 4952 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 23:47:47.063941 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:47.063913 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k" event={"ID":"d534649f-0944-4efd-b936-069af23b30ba","Type":"ContainerStarted","Data":"fae9a2a4b25c4ff64351ed6433d2c62cc856f9a9595c8413f7e2948f06d65ec2"} Oct 30 23:47:48.081442 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:48.081399 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k" event={"ID":"d534649f-0944-4efd-b936-069af23b30ba","Type":"ContainerStarted","Data":"af1d771f0be7e6dcada89dac0a154b5b1217edd731ce0ef30bc3864530280c95"} Oct 30 23:47:48.081820 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:48.081501 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k" Oct 30 23:47:48.198056 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:48.197938 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k" podStartSLOduration=2.338175138 podStartE2EDuration="3.197925518s" totalImagesPullingTime="859.75038ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:47:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:47:48.195659877 +0000 UTC m=+2487.339490086" watchObservedRunningTime="2026-10-30 23:47:48.197925518 +0000 UTC m=+2487.341755722" Oct 30 23:47:49.090103 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:49.090069 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3c9613-75d9965b8c-g924m" podUID="ec586e23-efdd-4fd1-b54c-3c11b5d90ae1" containerName="payload-pre-processing" containerID="cri-o://4c7cd9678ca6c652205f11c659e992ca3e0cf0915f2977379470fed7afb534be" gracePeriod=25 Oct 30 23:47:50.980767 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:50.980732 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:47:51.142565 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:47:51.142529 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:47:52.841140 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:52.841106 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" podUID="86a57a21-72bc-4ce4-8d1e-bf29214ad4de" containerName="main" containerID="cri-o://96c59deefb3e7054f39ff5f6e62ce4ea81d772b539af4e5f248c5b06aa93b927" gracePeriod=45 Oct 30 23:47:53.102563 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:53.102500 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" Oct 30 23:47:53.114095 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:53.114066 4952 generic.go:356] "Generic (PLEG): container finished" podID="86a57a21-72bc-4ce4-8d1e-bf29214ad4de" containerID="96c59deefb3e7054f39ff5f6e62ce4ea81d772b539af4e5f248c5b06aa93b927" exitCode=0 Oct 30 23:47:53.114235 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:53.114091 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" Oct 30 23:47:53.114235 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:53.114137 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" event={"ID":"86a57a21-72bc-4ce4-8d1e-bf29214ad4de","Type":"ContainerDied","Data":"96c59deefb3e7054f39ff5f6e62ce4ea81d772b539af4e5f248c5b06aa93b927"} Oct 30 23:47:53.114235 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:53.114173 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt" event={"ID":"86a57a21-72bc-4ce4-8d1e-bf29214ad4de","Type":"ContainerDied","Data":"a0598d54525aae44e083da581ebce15854b7743c496946c00f5952584ceee168"} Oct 30 23:47:53.114235 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:53.114191 4952 scope.go:122] "RemoveContainer" containerID="96c59deefb3e7054f39ff5f6e62ce4ea81d772b539af4e5f248c5b06aa93b927" Oct 30 23:47:53.123962 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:53.123945 4952 scope.go:122] "RemoveContainer" containerID="96c59deefb3e7054f39ff5f6e62ce4ea81d772b539af4e5f248c5b06aa93b927" Oct 30 23:47:53.124279 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:47:53.124261 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96c59deefb3e7054f39ff5f6e62ce4ea81d772b539af4e5f248c5b06aa93b927\": container with ID starting with 96c59deefb3e7054f39ff5f6e62ce4ea81d772b539af4e5f248c5b06aa93b927 not found: ID does not exist" containerID="96c59deefb3e7054f39ff5f6e62ce4ea81d772b539af4e5f248c5b06aa93b927" Oct 30 23:47:53.124348 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:53.124287 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96c59deefb3e7054f39ff5f6e62ce4ea81d772b539af4e5f248c5b06aa93b927"} err="failed to get container status \"96c59deefb3e7054f39ff5f6e62ce4ea81d772b539af4e5f248c5b06aa93b927\": rpc error: code = NotFound desc = could not find container \"96c59deefb3e7054f39ff5f6e62ce4ea81d772b539af4e5f248c5b06aa93b927\": container with ID starting with 96c59deefb3e7054f39ff5f6e62ce4ea81d772b539af4e5f248c5b06aa93b927 not found: ID does not exist" Oct 30 23:47:53.212884 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:53.212863 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-model-cache\") pod \"86a57a21-72bc-4ce4-8d1e-bf29214ad4de\" (UID: \"86a57a21-72bc-4ce4-8d1e-bf29214ad4de\") " Oct 30 23:47:53.212992 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:53.212919 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-kube-api-access-kmx65\" (UniqueName: \"kubernetes.io/projected/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-kube-api-access-kmx65\") pod \"86a57a21-72bc-4ce4-8d1e-bf29214ad4de\" (UID: \"86a57a21-72bc-4ce4-8d1e-bf29214ad4de\") " Oct 30 23:47:53.213063 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:53.213008 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-dshm\" (UniqueName: \"kubernetes.io/empty-dir/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-dshm\") pod \"86a57a21-72bc-4ce4-8d1e-bf29214ad4de\" (UID: \"86a57a21-72bc-4ce4-8d1e-bf29214ad4de\") " Oct 30 23:47:53.213125 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:53.213082 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-tls-certs\" (UniqueName: \"kubernetes.io/secret/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-tls-certs\") pod \"86a57a21-72bc-4ce4-8d1e-bf29214ad4de\" (UID: \"86a57a21-72bc-4ce4-8d1e-bf29214ad4de\") " Oct 30 23:47:53.213179 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:53.213127 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-model-cache" pod "86a57a21-72bc-4ce4-8d1e-bf29214ad4de" (UID: "86a57a21-72bc-4ce4-8d1e-bf29214ad4de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:47:53.213179 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:53.213150 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-home\" (UniqueName: \"kubernetes.io/empty-dir/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-home\") pod \"86a57a21-72bc-4ce4-8d1e-bf29214ad4de\" (UID: \"86a57a21-72bc-4ce4-8d1e-bf29214ad4de\") " Oct 30 23:47:53.213260 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:53.213221 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-tmp-dir\") pod \"86a57a21-72bc-4ce4-8d1e-bf29214ad4de\" (UID: \"86a57a21-72bc-4ce4-8d1e-bf29214ad4de\") " Oct 30 23:47:53.213420 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:53.213396 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-home" pod "86a57a21-72bc-4ce4-8d1e-bf29214ad4de" (UID: "86a57a21-72bc-4ce4-8d1e-bf29214ad4de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:47:53.213491 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:53.213475 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-tmp-dir" pod "86a57a21-72bc-4ce4-8d1e-bf29214ad4de" (UID: "86a57a21-72bc-4ce4-8d1e-bf29214ad4de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:47:53.213703 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:53.213678 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-home\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:47:53.213703 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:53.213698 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-tmp-dir\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:47:53.213703 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:53.213707 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-model-cache\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:47:53.215470 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:53.215445 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-kube-api-access-kmx65" pod "86a57a21-72bc-4ce4-8d1e-bf29214ad4de" (UID: "86a57a21-72bc-4ce4-8d1e-bf29214ad4de"). InnerVolumeSpecName "kube-api-access-kmx65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:47:53.215557 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:53.215473 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-tls-certs" pod "86a57a21-72bc-4ce4-8d1e-bf29214ad4de" (UID: "86a57a21-72bc-4ce4-8d1e-bf29214ad4de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:47:53.215557 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:53.215473 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-dshm" pod "86a57a21-72bc-4ce4-8d1e-bf29214ad4de" (UID: "86a57a21-72bc-4ce4-8d1e-bf29214ad4de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:47:53.314923 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:53.314894 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-dshm\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:47:53.314923 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:53.314921 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-tls-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:47:53.314923 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:53.314931 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmx65\" (UniqueName: \"kubernetes.io/projected/86a57a21-72bc-4ce4-8d1e-bf29214ad4de-kube-api-access-kmx65\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:47:53.536489 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:53.536455 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt"] Oct 30 23:47:53.542181 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:53.542155 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-3c9613/distinct-w2-3c9613-kserve-6b4fc979f4-zwmlt"] Oct 30 23:47:54.093875 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:54.093849 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k" Oct 30 23:47:54.373674 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:54.373596 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3c9613-75d9965b8c-g924m" podUID="ec586e23-efdd-4fd1-b54c-3c11b5d90ae1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 23:47:54.970690 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:54.970654 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86a57a21-72bc-4ce4-8d1e-bf29214ad4de" path="/var/lib/kubelet/pods/86a57a21-72bc-4ce4-8d1e-bf29214ad4de/volumes" Oct 30 23:47:55.776809 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:47:55.776772 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:48:00.512520 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:00.512443 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:48:00.738896 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:00.738845 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:48:04.374218 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:04.374179 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3c9613-75d9965b8c-g924m" podUID="ec586e23-efdd-4fd1-b54c-3c11b5d90ae1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 23:48:05.149639 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:05.149605 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w"] Oct 30 23:48:05.150132 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:05.150119 4952 cpu_manager.go:414] "containerMap: removing container" podUID="86a57a21-72bc-4ce4-8d1e-bf29214ad4de" containerName="main" Oct 30 23:48:05.150189 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:05.150133 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86a57a21-72bc-4ce4-8d1e-bf29214ad4de" containerName="main" Oct 30 23:48:05.150253 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:05.150244 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="86a57a21-72bc-4ce4-8d1e-bf29214ad4de" containerName="main" Oct 30 23:48:05.154703 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:05.154682 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w" Oct 30 23:48:05.207689 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:05.207659 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-a8204d-serving-cert\"" Oct 30 23:48:05.245698 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:05.245662 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccd7384b-d043-4f0f-b4e8-efdcea747317-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w\" (UID: \"ccd7384b-d043-4f0f-b4e8-efdcea747317\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w" Oct 30 23:48:05.245877 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:05.245724 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccd7384b-d043-4f0f-b4e8-efdcea747317-metrics-certs\") pod \"maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w\" (UID: \"ccd7384b-d043-4f0f-b4e8-efdcea747317\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w" Oct 30 23:48:05.245877 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:05.245853 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl2fv\" (UniqueName: \"kubernetes.io/projected/ccd7384b-d043-4f0f-b4e8-efdcea747317-kube-api-access-nl2fv\") pod \"maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w\" (UID: \"ccd7384b-d043-4f0f-b4e8-efdcea747317\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w" Oct 30 23:48:05.246057 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:05.245983 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccd7384b-d043-4f0f-b4e8-efdcea747317-maas-api-tls\") pod \"maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w\" (UID: \"ccd7384b-d043-4f0f-b4e8-efdcea747317\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w" Oct 30 23:48:05.261392 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:05.261366 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w"] Oct 30 23:48:05.347609 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:05.347570 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccd7384b-d043-4f0f-b4e8-efdcea747317-maas-api-tls\") pod \"maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w\" (UID: \"ccd7384b-d043-4f0f-b4e8-efdcea747317\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w" Oct 30 23:48:05.347772 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:05.347667 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccd7384b-d043-4f0f-b4e8-efdcea747317-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w\" (UID: \"ccd7384b-d043-4f0f-b4e8-efdcea747317\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w" Oct 30 23:48:05.347772 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:05.347714 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccd7384b-d043-4f0f-b4e8-efdcea747317-metrics-certs\") pod \"maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w\" (UID: \"ccd7384b-d043-4f0f-b4e8-efdcea747317\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w" Oct 30 23:48:05.347772 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:48:05.347717 4952 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-a8204d-serving-cert: secret "maas-api-e2e-inf-b-a8204d-serving-cert" not found Oct 30 23:48:05.347772 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:05.347746 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl2fv\" (UniqueName: \"kubernetes.io/projected/ccd7384b-d043-4f0f-b4e8-efdcea747317-kube-api-access-nl2fv\") pod \"maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w\" (UID: \"ccd7384b-d043-4f0f-b4e8-efdcea747317\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w" Oct 30 23:48:05.348017 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:48:05.347795 4952 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccd7384b-d043-4f0f-b4e8-efdcea747317-maas-api-tls podName:ccd7384b-d043-4f0f-b4e8-efdcea747317 nodeName:}" failed. No retries permitted until 2026-10-30 23:48:05.847779517 +0000 UTC m=+2504.991609713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ccd7384b-d043-4f0f-b4e8-efdcea747317-maas-api-tls") pod "maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w" (UID: "ccd7384b-d043-4f0f-b4e8-efdcea747317") : secret "maas-api-e2e-inf-b-a8204d-serving-cert" not found Oct 30 23:48:05.348368 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:05.348345 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccd7384b-d043-4f0f-b4e8-efdcea747317-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w\" (UID: \"ccd7384b-d043-4f0f-b4e8-efdcea747317\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w" Oct 30 23:48:05.350444 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:05.350425 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccd7384b-d043-4f0f-b4e8-efdcea747317-metrics-certs\") pod \"maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w\" (UID: \"ccd7384b-d043-4f0f-b4e8-efdcea747317\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w" Oct 30 23:48:05.412777 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:05.412704 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl2fv\" (UniqueName: \"kubernetes.io/projected/ccd7384b-d043-4f0f-b4e8-efdcea747317-kube-api-access-nl2fv\") pod \"maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w\" (UID: \"ccd7384b-d043-4f0f-b4e8-efdcea747317\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w" Oct 30 23:48:05.455659 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:05.455622 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:48:05.671741 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:05.671644 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:48:05.854735 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:05.854702 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccd7384b-d043-4f0f-b4e8-efdcea747317-maas-api-tls\") pod \"maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w\" (UID: \"ccd7384b-d043-4f0f-b4e8-efdcea747317\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w" Oct 30 23:48:05.857670 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:05.857643 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccd7384b-d043-4f0f-b4e8-efdcea747317-maas-api-tls\") pod \"maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w\" (UID: \"ccd7384b-d043-4f0f-b4e8-efdcea747317\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w" Oct 30 23:48:06.067658 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:06.067593 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w" Oct 30 23:48:06.446632 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:06.446603 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w"] Oct 30 23:48:06.447145 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:48:06.447119 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccd7384b_d043_4f0f_b4e8_efdcea747317.slice/crio-80a5ec38e4fa5861aad1e35b16d28ee3a4adeedb772d205163b8ce36e0e3eb5d WatchSource:0}: Error finding container 80a5ec38e4fa5861aad1e35b16d28ee3a4adeedb772d205163b8ce36e0e3eb5d: Status 404 returned error can't find the container with id 80a5ec38e4fa5861aad1e35b16d28ee3a4adeedb772d205163b8ce36e0e3eb5d Oct 30 23:48:07.205323 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:07.205285 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w" event={"ID":"ccd7384b-d043-4f0f-b4e8-efdcea747317","Type":"ContainerStarted","Data":"80a5ec38e4fa5861aad1e35b16d28ee3a4adeedb772d205163b8ce36e0e3eb5d"} Oct 30 23:48:08.232957 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:08.232906 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w" event={"ID":"ccd7384b-d043-4f0f-b4e8-efdcea747317","Type":"ContainerStarted","Data":"034028d63180aad49b8f11794139321d61a33044fac9487e1cfdc10e616c227d"} Oct 30 23:48:08.233528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:08.233030 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w" Oct 30 23:48:08.371963 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:08.371894 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w" podStartSLOduration=2.5363542580000003 podStartE2EDuration="3.37187634s" totalImagesPullingTime="835.522082ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:48:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:48:08.369952758 +0000 UTC m=+2507.513782965" watchObservedRunningTime="2026-10-30 23:48:08.37187634 +0000 UTC m=+2507.515706547" Oct 30 23:48:08.420615 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:08.420583 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:48:09.243969 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:09.243938 4952 generic.go:356] "Generic (PLEG): container finished" podID="ec586e23-efdd-4fd1-b54c-3c11b5d90ae1" containerID="4c7cd9678ca6c652205f11c659e992ca3e0cf0915f2977379470fed7afb534be" exitCode=0 Oct 30 23:48:09.244347 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:09.244011 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3c9613-75d9965b8c-g924m" event={"ID":"ec586e23-efdd-4fd1-b54c-3c11b5d90ae1","Type":"ContainerDied","Data":"4c7cd9678ca6c652205f11c659e992ca3e0cf0915f2977379470fed7afb534be"} Oct 30 23:48:09.354928 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:09.354874 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3c9613-75d9965b8c-g924m" Oct 30 23:48:09.404718 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:09.404682 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec586e23-efdd-4fd1-b54c-3c11b5d90ae1-kube-api-access-srf55\" (UniqueName: \"kubernetes.io/projected/ec586e23-efdd-4fd1-b54c-3c11b5d90ae1-kube-api-access-srf55\") pod \"ec586e23-efdd-4fd1-b54c-3c11b5d90ae1\" (UID: \"ec586e23-efdd-4fd1-b54c-3c11b5d90ae1\") " Oct 30 23:48:09.404865 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:09.404845 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec586e23-efdd-4fd1-b54c-3c11b5d90ae1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec586e23-efdd-4fd1-b54c-3c11b5d90ae1-plugins-config-volume\") pod \"ec586e23-efdd-4fd1-b54c-3c11b5d90ae1\" (UID: \"ec586e23-efdd-4fd1-b54c-3c11b5d90ae1\") " Oct 30 23:48:09.405245 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:09.405223 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec586e23-efdd-4fd1-b54c-3c11b5d90ae1-plugins-config-volume" pod "ec586e23-efdd-4fd1-b54c-3c11b5d90ae1" (UID: "ec586e23-efdd-4fd1-b54c-3c11b5d90ae1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:48:09.407332 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:09.407306 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec586e23-efdd-4fd1-b54c-3c11b5d90ae1-kube-api-access-srf55" pod "ec586e23-efdd-4fd1-b54c-3c11b5d90ae1" (UID: "ec586e23-efdd-4fd1-b54c-3c11b5d90ae1"). InnerVolumeSpecName "kube-api-access-srf55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:48:09.506480 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:09.506424 4952 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec586e23-efdd-4fd1-b54c-3c11b5d90ae1-plugins-config-volume\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:48:09.506480 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:09.506454 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srf55\" (UniqueName: \"kubernetes.io/projected/ec586e23-efdd-4fd1-b54c-3c11b5d90ae1-kube-api-access-srf55\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:48:10.252264 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:10.252233 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3c9613-75d9965b8c-g924m" Oct 30 23:48:10.252760 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:10.252271 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3c9613-75d9965b8c-g924m" event={"ID":"ec586e23-efdd-4fd1-b54c-3c11b5d90ae1","Type":"ContainerDied","Data":"cdd1f80ec91ce3241ef250d55a9d3accbcd2ef5803d3fa34cbff6928580ffa6e"} Oct 30 23:48:10.252760 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:10.252329 4952 scope.go:122] "RemoveContainer" containerID="4c7cd9678ca6c652205f11c659e992ca3e0cf0915f2977379470fed7afb534be" Oct 30 23:48:10.416487 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:10.416459 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3c9613-75d9965b8c-g924m"] Oct 30 23:48:10.465595 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:10.465571 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3c9613-75d9965b8c-g924m"] Oct 30 23:48:10.676415 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:10.676385 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:48:10.972897 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:10.972821 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec586e23-efdd-4fd1-b54c-3c11b5d90ae1" path="/var/lib/kubelet/pods/ec586e23-efdd-4fd1-b54c-3c11b5d90ae1/volumes" Oct 30 23:48:14.250506 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:14.250479 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w" Oct 30 23:48:14.591075 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:14.590999 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:48:14.829211 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:14.829174 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:48:15.494874 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:15.494839 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:48:15.806442 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:15.806344 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:48:20.320304 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:20.320251 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:48:21.141467 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:48:21.141430 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:48:22.090387 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:22.090345 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:48:22.334728 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:22.334651 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:48:23.120189 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:23.120155 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:48:23.350813 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:23.350779 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:48:24.857207 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:24.857172 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:48:25.181595 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:25.181506 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:48:30.165029 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:30.164997 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:48:34.992934 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:34.992902 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:48:39.728543 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:39.728510 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:48:44.563198 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:44.563166 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:48:49.441902 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:49.441863 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:48:51.143552 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:48:51.143516 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:48:52.987107 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:52.987074 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:48:54.491696 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:54.491665 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:48:54.725088 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:54.725029 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:48:56.685953 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:56.685916 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:48:59.696805 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:48:59.696766 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:03.630225 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:03.630187 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:04.959470 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:04.959438 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:08.085870 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:08.084313 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:08.313610 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:08.313575 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:09.016539 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:09.016501 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:10.358077 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:10.358005 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:15.671556 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:15.671511 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:21.118471 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:21.118414 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:21.143504 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:49:21.143469 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:49:21.346968 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:21.346930 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:21.693960 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:21.693926 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:22.411240 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:22.411209 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:22.886197 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:22.886154 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:25.199230 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:25.199192 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:25.421457 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:25.421422 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:25.848637 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:25.848593 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:26.326886 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:26.326792 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:49:26.577127 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:26.575027 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:27.086749 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:27.086705 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:27.357417 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:27.357337 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:27.842730 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:27.842698 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:28.074223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:28.074184 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:28.300629 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:28.300530 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:28.931732 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:28.931699 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:30.488056 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:30.487172 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:30.721623 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:30.721576 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:33.123105 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:33.123026 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:35.018444 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:35.018409 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:35.332701 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:35.332586 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:35.367150 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:35.367116 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs"] Oct 30 23:49:35.367328 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:35.367307 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" podUID="bf7e392e-f85e-4e50-8b1c-c0423a05a38c" containerName="istio-proxy" containerID="cri-o://8fab35f8572f4b75affb68e6d7bf7ac7724dea345535f431c8d6e30543c43f2d" gracePeriod=30 Oct 30 23:49:35.587095 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:35.587022 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:36.464322 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:36.464282 4952 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 23:49:36.464705 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:36.464358 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" podUID="bf7e392e-f85e-4e50-8b1c-c0423a05a38c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:49:38.366262 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:38.366223 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:40.636120 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:40.636064 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" Oct 30 23:49:40.671180 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:40.671158 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-istio-data\") pod \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " Oct 30 23:49:40.671300 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:40.671197 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-istio-envoy\") pod \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " Oct 30 23:49:40.671300 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:40.671236 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-istio-podinfo\") pod \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " Oct 30 23:49:40.671300 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:40.671270 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-kube-api-access-8k4mx\" (UniqueName: \"kubernetes.io/projected/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-kube-api-access-8k4mx\") pod \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " Oct 30 23:49:40.671452 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:40.671301 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-workload-socket\") pod \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " Oct 30 23:49:40.671452 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:40.671371 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-istio-token\" (UniqueName: \"kubernetes.io/projected/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-istio-token\") pod \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " Oct 30 23:49:40.671570 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:40.671442 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-istio-data" pod "bf7e392e-f85e-4e50-8b1c-c0423a05a38c" (UID: "bf7e392e-f85e-4e50-8b1c-c0423a05a38c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:40.671570 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:40.671456 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-credential-socket\") pod \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " Oct 30 23:49:40.671683 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:40.671571 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-istiod-ca-cert\") pod \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " Oct 30 23:49:40.671683 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:40.671638 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-workload-certs\") pod \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\" (UID: \"bf7e392e-f85e-4e50-8b1c-c0423a05a38c\") " Oct 30 23:49:40.671683 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:40.671654 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-workload-socket" pod "bf7e392e-f85e-4e50-8b1c-c0423a05a38c" (UID: "bf7e392e-f85e-4e50-8b1c-c0423a05a38c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:40.671859 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:40.671722 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-credential-socket" pod "bf7e392e-f85e-4e50-8b1c-c0423a05a38c" (UID: "bf7e392e-f85e-4e50-8b1c-c0423a05a38c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:40.672005 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:40.671981 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-workload-certs" pod "bf7e392e-f85e-4e50-8b1c-c0423a05a38c" (UID: "bf7e392e-f85e-4e50-8b1c-c0423a05a38c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:40.672289 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:40.672008 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-istiod-ca-cert" pod "bf7e392e-f85e-4e50-8b1c-c0423a05a38c" (UID: "bf7e392e-f85e-4e50-8b1c-c0423a05a38c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:49:40.672496 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:40.672472 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-workload-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:49:40.672587 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:40.672501 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-credential-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:49:40.672587 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:40.672519 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-istiod-ca-cert\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:49:40.672587 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:40.672535 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-workload-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:49:40.672587 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:40.672550 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-istio-data\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:49:40.673823 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:40.673798 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-istio-podinfo" pod "bf7e392e-f85e-4e50-8b1c-c0423a05a38c" (UID: "bf7e392e-f85e-4e50-8b1c-c0423a05a38c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:49:40.673909 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:40.673857 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-kube-api-access-8k4mx" pod "bf7e392e-f85e-4e50-8b1c-c0423a05a38c" (UID: "bf7e392e-f85e-4e50-8b1c-c0423a05a38c"). InnerVolumeSpecName "kube-api-access-8k4mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:49:40.674074 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:40.674048 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-istio-token" pod "bf7e392e-f85e-4e50-8b1c-c0423a05a38c" (UID: "bf7e392e-f85e-4e50-8b1c-c0423a05a38c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:49:40.674074 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:40.674053 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-istio-envoy" pod "bf7e392e-f85e-4e50-8b1c-c0423a05a38c" (UID: "bf7e392e-f85e-4e50-8b1c-c0423a05a38c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:49:40.774168 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:40.774136 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-istio-token\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:49:40.774168 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:40.774166 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-istio-envoy\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:49:40.774360 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:40.774181 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-istio-podinfo\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:49:40.774360 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:40.774196 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k4mx\" (UniqueName: \"kubernetes.io/projected/bf7e392e-f85e-4e50-8b1c-c0423a05a38c-kube-api-access-8k4mx\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:49:40.861354 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:40.861318 4952 generic.go:356] "Generic (PLEG): container finished" podID="bf7e392e-f85e-4e50-8b1c-c0423a05a38c" containerID="8fab35f8572f4b75affb68e6d7bf7ac7724dea345535f431c8d6e30543c43f2d" exitCode=0 Oct 30 23:49:40.861532 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:40.861370 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" event={"ID":"bf7e392e-f85e-4e50-8b1c-c0423a05a38c","Type":"ContainerDied","Data":"8fab35f8572f4b75affb68e6d7bf7ac7724dea345535f431c8d6e30543c43f2d"} Oct 30 23:49:40.861532 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:40.861419 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" event={"ID":"bf7e392e-f85e-4e50-8b1c-c0423a05a38c","Type":"ContainerDied","Data":"53e0aa872dde246fe012602e7e91c8b670889f6358ef48d4c57a39a4aea4e195"} Oct 30 23:49:40.861532 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:40.861446 4952 scope.go:122] "RemoveContainer" containerID="8fab35f8572f4b75affb68e6d7bf7ac7724dea345535f431c8d6e30543c43f2d" Oct 30 23:49:40.861532 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:40.861373 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs" Oct 30 23:49:40.874498 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:40.874478 4952 scope.go:122] "RemoveContainer" containerID="8fab35f8572f4b75affb68e6d7bf7ac7724dea345535f431c8d6e30543c43f2d" Oct 30 23:49:40.874834 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:49:40.874815 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fab35f8572f4b75affb68e6d7bf7ac7724dea345535f431c8d6e30543c43f2d\": container with ID starting with 8fab35f8572f4b75affb68e6d7bf7ac7724dea345535f431c8d6e30543c43f2d not found: ID does not exist" containerID="8fab35f8572f4b75affb68e6d7bf7ac7724dea345535f431c8d6e30543c43f2d" Oct 30 23:49:40.874901 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:40.874848 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fab35f8572f4b75affb68e6d7bf7ac7724dea345535f431c8d6e30543c43f2d"} err="failed to get container status \"8fab35f8572f4b75affb68e6d7bf7ac7724dea345535f431c8d6e30543c43f2d\": rpc error: code = NotFound desc = could not find container \"8fab35f8572f4b75affb68e6d7bf7ac7724dea345535f431c8d6e30543c43f2d\": container with ID starting with 8fab35f8572f4b75affb68e6d7bf7ac7724dea345535f431c8d6e30543c43f2d not found: ID does not exist" Oct 30 23:49:41.027642 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:41.027462 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs"] Oct 30 23:49:41.035218 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:41.035171 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-jfkzs"] Oct 30 23:49:42.973058 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:42.973005 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf7e392e-f85e-4e50-8b1c-c0423a05a38c" path="/var/lib/kubelet/pods/bf7e392e-f85e-4e50-8b1c-c0423a05a38c/volumes" Oct 30 23:49:43.254637 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:43.254521 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:45.755697 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:45.755644 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:48.081808 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:48.081760 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:51.141615 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:49:51.141577 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:49:52.877383 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:52.877329 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:49:57.694662 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:49:57.694622 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:50:00.275500 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:00.275450 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:50:02.485327 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:02.485287 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:50:06.455068 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:06.455011 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq"] Oct 30 23:50:06.455508 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:06.455493 4952 cpu_manager.go:414] "containerMap: removing container" podUID="ec586e23-efdd-4fd1-b54c-3c11b5d90ae1" containerName="payload-pre-processing" Oct 30 23:50:06.455556 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:06.455509 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec586e23-efdd-4fd1-b54c-3c11b5d90ae1" containerName="payload-pre-processing" Oct 30 23:50:06.455556 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:06.455522 4952 cpu_manager.go:414] "containerMap: removing container" podUID="bf7e392e-f85e-4e50-8b1c-c0423a05a38c" containerName="istio-proxy" Oct 30 23:50:06.455556 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:06.455529 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf7e392e-f85e-4e50-8b1c-c0423a05a38c" containerName="istio-proxy" Oct 30 23:50:06.455664 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:06.455655 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec586e23-efdd-4fd1-b54c-3c11b5d90ae1" containerName="payload-pre-processing" Oct 30 23:50:06.455699 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:06.455667 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf7e392e-f85e-4e50-8b1c-c0423a05a38c" containerName="istio-proxy" Oct 30 23:50:06.459198 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:06.459174 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq" Oct 30 23:50:06.518090 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:06.518057 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq"] Oct 30 23:50:06.562472 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:06.562424 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-71a68a\"/\"default-dockercfg-v55m5\"" Oct 30 23:50:06.562472 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:06.562472 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-71a68a\"/\"openshift-service-ca.crt\"" Oct 30 23:50:06.562472 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:06.562486 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-71a68a\"/\"test-model-71a68a-kserve-self-signed-certs\"" Oct 30 23:50:06.562720 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:06.562489 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-71a68a\"/\"kube-root-ca.crt\"" Oct 30 23:50:06.623413 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:06.623372 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn6sg\" (UniqueName: \"kubernetes.io/projected/39ddf73a-4de1-404a-8a9a-276804aa9209-kube-api-access-sn6sg\") pod \"test-model-71a68a-kserve-78ff84d5b7-5srmq\" (UID: \"39ddf73a-4de1-404a-8a9a-276804aa9209\") " pod="ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq" Oct 30 23:50:06.623624 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:06.623527 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39ddf73a-4de1-404a-8a9a-276804aa9209-tls-certs\") pod \"test-model-71a68a-kserve-78ff84d5b7-5srmq\" (UID: \"39ddf73a-4de1-404a-8a9a-276804aa9209\") " pod="ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq" Oct 30 23:50:06.623624 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:06.623580 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ddf73a-4de1-404a-8a9a-276804aa9209-dshm\") pod \"test-model-71a68a-kserve-78ff84d5b7-5srmq\" (UID: \"39ddf73a-4de1-404a-8a9a-276804aa9209\") " pod="ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq" Oct 30 23:50:06.623749 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:06.623637 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ddf73a-4de1-404a-8a9a-276804aa9209-tmp-dir\") pod \"test-model-71a68a-kserve-78ff84d5b7-5srmq\" (UID: \"39ddf73a-4de1-404a-8a9a-276804aa9209\") " pod="ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq" Oct 30 23:50:06.623749 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:06.623670 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ddf73a-4de1-404a-8a9a-276804aa9209-model-cache\") pod \"test-model-71a68a-kserve-78ff84d5b7-5srmq\" (UID: \"39ddf73a-4de1-404a-8a9a-276804aa9209\") " pod="ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq" Oct 30 23:50:06.623749 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:06.623712 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39ddf73a-4de1-404a-8a9a-276804aa9209-home\") pod \"test-model-71a68a-kserve-78ff84d5b7-5srmq\" (UID: \"39ddf73a-4de1-404a-8a9a-276804aa9209\") " pod="ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq" Oct 30 23:50:06.627840 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:06.627803 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:50:06.724723 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:06.724636 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ddf73a-4de1-404a-8a9a-276804aa9209-tmp-dir\") pod \"test-model-71a68a-kserve-78ff84d5b7-5srmq\" (UID: \"39ddf73a-4de1-404a-8a9a-276804aa9209\") " pod="ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq" Oct 30 23:50:06.724723 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:06.724687 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ddf73a-4de1-404a-8a9a-276804aa9209-model-cache\") pod \"test-model-71a68a-kserve-78ff84d5b7-5srmq\" (UID: \"39ddf73a-4de1-404a-8a9a-276804aa9209\") " pod="ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq" Oct 30 23:50:06.724930 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:06.724729 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39ddf73a-4de1-404a-8a9a-276804aa9209-home\") pod \"test-model-71a68a-kserve-78ff84d5b7-5srmq\" (UID: \"39ddf73a-4de1-404a-8a9a-276804aa9209\") " pod="ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq" Oct 30 23:50:06.724930 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:06.724768 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn6sg\" (UniqueName: \"kubernetes.io/projected/39ddf73a-4de1-404a-8a9a-276804aa9209-kube-api-access-sn6sg\") pod \"test-model-71a68a-kserve-78ff84d5b7-5srmq\" (UID: \"39ddf73a-4de1-404a-8a9a-276804aa9209\") " pod="ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq" Oct 30 23:50:06.725008 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:06.724935 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39ddf73a-4de1-404a-8a9a-276804aa9209-tls-certs\") pod \"test-model-71a68a-kserve-78ff84d5b7-5srmq\" (UID: \"39ddf73a-4de1-404a-8a9a-276804aa9209\") " pod="ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq" Oct 30 23:50:06.725008 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:06.724983 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ddf73a-4de1-404a-8a9a-276804aa9209-dshm\") pod \"test-model-71a68a-kserve-78ff84d5b7-5srmq\" (UID: \"39ddf73a-4de1-404a-8a9a-276804aa9209\") " pod="ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq" Oct 30 23:50:06.725151 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:06.725127 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ddf73a-4de1-404a-8a9a-276804aa9209-tmp-dir\") pod \"test-model-71a68a-kserve-78ff84d5b7-5srmq\" (UID: \"39ddf73a-4de1-404a-8a9a-276804aa9209\") " pod="ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq" Oct 30 23:50:06.725337 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:06.725136 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ddf73a-4de1-404a-8a9a-276804aa9209-model-cache\") pod \"test-model-71a68a-kserve-78ff84d5b7-5srmq\" (UID: \"39ddf73a-4de1-404a-8a9a-276804aa9209\") " pod="ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq" Oct 30 23:50:06.725627 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:06.725601 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39ddf73a-4de1-404a-8a9a-276804aa9209-home\") pod \"test-model-71a68a-kserve-78ff84d5b7-5srmq\" (UID: \"39ddf73a-4de1-404a-8a9a-276804aa9209\") " pod="ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq" Oct 30 23:50:06.727529 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:06.727503 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ddf73a-4de1-404a-8a9a-276804aa9209-dshm\") pod \"test-model-71a68a-kserve-78ff84d5b7-5srmq\" (UID: \"39ddf73a-4de1-404a-8a9a-276804aa9209\") " pod="ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq" Oct 30 23:50:06.727828 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:06.727808 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39ddf73a-4de1-404a-8a9a-276804aa9209-tls-certs\") pod \"test-model-71a68a-kserve-78ff84d5b7-5srmq\" (UID: \"39ddf73a-4de1-404a-8a9a-276804aa9209\") " pod="ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq" Oct 30 23:50:06.783142 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:06.783110 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn6sg\" (UniqueName: \"kubernetes.io/projected/39ddf73a-4de1-404a-8a9a-276804aa9209-kube-api-access-sn6sg\") pod \"test-model-71a68a-kserve-78ff84d5b7-5srmq\" (UID: \"39ddf73a-4de1-404a-8a9a-276804aa9209\") " pod="ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq" Oct 30 23:50:07.071900 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:07.071807 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq" Oct 30 23:50:07.261660 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:07.261622 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:50:07.452583 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:07.452557 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq"] Oct 30 23:50:07.452982 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:50:07.452961 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39ddf73a_4de1_404a_8a9a_276804aa9209.slice/crio-ac6241d5b8d106c9f1a9e3537cfea960fff3fb9da828497d85079861284e74ae WatchSource:0}: Error finding container ac6241d5b8d106c9f1a9e3537cfea960fff3fb9da828497d85079861284e74ae: Status 404 returned error can't find the container with id ac6241d5b8d106c9f1a9e3537cfea960fff3fb9da828497d85079861284e74ae Oct 30 23:50:08.042812 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:08.042767 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq" event={"ID":"39ddf73a-4de1-404a-8a9a-276804aa9209","Type":"ContainerStarted","Data":"d4eec5bb1d2355560ecfcf0977b4615951cb6fbdbf7c05b784c4597b0c69ab12"} Oct 30 23:50:08.042812 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:08.042814 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq" event={"ID":"39ddf73a-4de1-404a-8a9a-276804aa9209","Type":"ContainerStarted","Data":"ac6241d5b8d106c9f1a9e3537cfea960fff3fb9da828497d85079861284e74ae"} Oct 30 23:50:08.072298 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:08.072257 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq" Oct 30 23:50:08.158896 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:08.158853 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq" podStartSLOduration=2.158841166 podStartE2EDuration="2.158841166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:50:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:50:08.157617651 +0000 UTC m=+2627.301447869" watchObservedRunningTime="2026-10-30 23:50:08.158841166 +0000 UTC m=+2627.302671371" Oct 30 23:50:09.408747 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:09.408715 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:50:12.061940 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:12.061890 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:50:16.828750 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:16.828716 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:50:17.072565 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:17.072531 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq" Oct 30 23:50:17.085271 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:17.085203 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq" Oct 30 23:50:17.112244 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:17.112222 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq" Oct 30 23:50:21.142189 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:50:21.142155 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:50:21.597449 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:21.597409 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:50:25.540598 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:25.540559 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:50:26.377909 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:26.377867 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:50:27.280254 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:27.280217 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:50:31.128530 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:31.128449 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:50:31.349807 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:31.349778 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:50:31.582537 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:31.582455 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:50:36.059172 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:36.059140 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:50:40.864934 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:40.864888 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:50:45.768160 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:45.768117 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:50:50.589178 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:50.589148 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:50:51.141773 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:50:51.141731 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:50:54.749154 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:54.749088 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:50:56.269995 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:56.269959 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:50:56.496025 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:56.495985 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:50:56.729418 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:56.729384 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:50:57.203786 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:50:57.203740 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:01.578893 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:51:01.578851 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:03.015754 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:51:03.015708 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:03.252764 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:51:03.252733 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:03.475810 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:51:03.475779 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:06.632232 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:51:06.632200 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:09.967364 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:51:09.967331 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:51:11.616061 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:51:11.615550 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:16.184688 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:51:16.184648 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:16.411138 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:51:16.411082 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:21.148382 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:51:21.148341 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:51:21.168593 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:51:21.168560 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:21.401712 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:51:21.401645 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft_91651ce2-4673-4ced-b26d-e94ff8829d11/extract/0.log" Oct 30 23:51:21.412258 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:51:21.412225 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/2.log" Oct 30 23:51:21.416363 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:51:21.416346 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft_91651ce2-4673-4ced-b26d-e94ff8829d11/extract/0.log" Oct 30 23:51:21.418215 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:51:21.418193 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 30 23:51:21.426861 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:51:21.426843 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/2.log" Oct 30 23:51:21.432568 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:51:21.432552 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 30 23:51:25.878098 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:51:25.878067 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:26.094956 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:51:26.094927 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:30.767150 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:51:30.767088 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:35.534925 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:51:35.534875 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:37.995957 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:51:37.995921 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:51:40.269761 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:51:40.269720 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:45.138482 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:51:45.138441 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:50.021727 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:51:50.021619 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:50.241912 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:51:50.241881 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:51.142728 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:51:51.142690 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:51:53.285869 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:51:53.285833 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:53.516696 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:51:53.516661 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:53.740965 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:51:53.740933 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:54.872602 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:51:54.872561 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:51:59.662668 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:51:59.662627 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:00.438187 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:00.438144 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:04.569760 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:04.569718 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:08.841363 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:08.841331 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4"] Oct 30 23:52:08.841917 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:08.841533 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" podUID="d75821f5-34cd-4eeb-a9cc-a0c19573f0f8" containerName="istio-proxy" containerID="cri-o://cff7a5c11e1ce818e4db267beceec569eb4bbe27eac34c76ce518b1b3eee9ae3" gracePeriod=30 Oct 30 23:52:08.891094 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:08.891057 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp"] Oct 30 23:52:08.891298 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:08.891276 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" podUID="553c8b63-e975-498f-9313-878fbf99066d" containerName="istio-proxy" containerID="cri-o://0e539ca1c5acab406e0fcffffeb55994cadd84ab227f590dadf31a310a3c588a" gracePeriod=30 Oct 30 23:52:09.459894 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:09.459840 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:10.174763 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:10.174731 4952 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 23:52:10.175190 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:10.174797 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" podUID="553c8b63-e975-498f-9313-878fbf99066d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:52:13.623027 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:13.622984 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:52:14.178771 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.178744 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" Oct 30 23:52:14.244114 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.243978 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" Oct 30 23:52:14.265910 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.265880 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:14.280874 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.280845 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-istio-envoy\") pod \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " Oct 30 23:52:14.281100 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.280884 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-istio-podinfo\") pod \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " Oct 30 23:52:14.281100 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.280907 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-workload-certs\") pod \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " Oct 30 23:52:14.281100 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.280957 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-istiod-ca-cert\") pod \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " Oct 30 23:52:14.281301 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.281108 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/553c8b63-e975-498f-9313-878fbf99066d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/553c8b63-e975-498f-9313-878fbf99066d-istio-envoy\") pod \"553c8b63-e975-498f-9313-878fbf99066d\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " Oct 30 23:52:14.281301 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.281171 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-kube-api-access-9kgzh\" (UniqueName: \"kubernetes.io/projected/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-kube-api-access-9kgzh\") pod \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " Oct 30 23:52:14.281301 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.281202 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/553c8b63-e975-498f-9313-878fbf99066d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/553c8b63-e975-498f-9313-878fbf99066d-istiod-ca-cert\") pod \"553c8b63-e975-498f-9313-878fbf99066d\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " Oct 30 23:52:14.281301 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.281253 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-workload-socket\") pod \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " Oct 30 23:52:14.281301 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.281294 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/553c8b63-e975-498f-9313-878fbf99066d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/553c8b63-e975-498f-9313-878fbf99066d-istio-data\") pod \"553c8b63-e975-498f-9313-878fbf99066d\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " Oct 30 23:52:14.281559 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.281346 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-istiod-ca-cert" pod "d75821f5-34cd-4eeb-a9cc-a0c19573f0f8" (UID: "d75821f5-34cd-4eeb-a9cc-a0c19573f0f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:52:14.281559 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.281377 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/553c8b63-e975-498f-9313-878fbf99066d-kube-api-access-spvwr\" (UniqueName: \"kubernetes.io/projected/553c8b63-e975-498f-9313-878fbf99066d-kube-api-access-spvwr\") pod \"553c8b63-e975-498f-9313-878fbf99066d\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " Oct 30 23:52:14.281559 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.281413 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/553c8b63-e975-498f-9313-878fbf99066d-istio-token\" (UniqueName: \"kubernetes.io/projected/553c8b63-e975-498f-9313-878fbf99066d-istio-token\") pod \"553c8b63-e975-498f-9313-878fbf99066d\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " Oct 30 23:52:14.281559 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.281464 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-istio-token\" (UniqueName: \"kubernetes.io/projected/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-istio-token\") pod \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " Oct 30 23:52:14.281559 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.281528 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-credential-socket\") pod \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " Oct 30 23:52:14.282096 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.281558 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/553c8b63-e975-498f-9313-878fbf99066d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/553c8b63-e975-498f-9313-878fbf99066d-credential-socket\") pod \"553c8b63-e975-498f-9313-878fbf99066d\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " Oct 30 23:52:14.282096 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.281607 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/553c8b63-e975-498f-9313-878fbf99066d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/553c8b63-e975-498f-9313-878fbf99066d-istio-podinfo\") pod \"553c8b63-e975-498f-9313-878fbf99066d\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " Oct 30 23:52:14.282096 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.281626 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-workload-socket" pod "d75821f5-34cd-4eeb-a9cc-a0c19573f0f8" (UID: "d75821f5-34cd-4eeb-a9cc-a0c19573f0f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:52:14.282096 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.281649 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/553c8b63-e975-498f-9313-878fbf99066d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/553c8b63-e975-498f-9313-878fbf99066d-workload-certs\") pod \"553c8b63-e975-498f-9313-878fbf99066d\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " Oct 30 23:52:14.282096 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.281706 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-istio-data\") pod \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\" (UID: \"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8\") " Oct 30 23:52:14.282096 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.281760 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/553c8b63-e975-498f-9313-878fbf99066d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/553c8b63-e975-498f-9313-878fbf99066d-workload-socket\") pod \"553c8b63-e975-498f-9313-878fbf99066d\" (UID: \"553c8b63-e975-498f-9313-878fbf99066d\") " Oct 30 23:52:14.282096 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.281847 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-credential-socket" pod "d75821f5-34cd-4eeb-a9cc-a0c19573f0f8" (UID: "d75821f5-34cd-4eeb-a9cc-a0c19573f0f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:52:14.282096 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.281985 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/553c8b63-e975-498f-9313-878fbf99066d-istiod-ca-cert" pod "553c8b63-e975-498f-9313-878fbf99066d" (UID: "553c8b63-e975-498f-9313-878fbf99066d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:52:14.282567 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.282325 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/553c8b63-e975-498f-9313-878fbf99066d-istio-data" pod "553c8b63-e975-498f-9313-878fbf99066d" (UID: "553c8b63-e975-498f-9313-878fbf99066d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:52:14.282567 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.282479 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-istiod-ca-cert\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:52:14.282567 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.282502 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/553c8b63-e975-498f-9313-878fbf99066d-istiod-ca-cert\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:52:14.282567 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.282517 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-workload-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:52:14.282567 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.282522 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/553c8b63-e975-498f-9313-878fbf99066d-workload-certs" pod "553c8b63-e975-498f-9313-878fbf99066d" (UID: "553c8b63-e975-498f-9313-878fbf99066d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:52:14.282567 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.282532 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/553c8b63-e975-498f-9313-878fbf99066d-istio-data\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:52:14.282567 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.282546 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-credential-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:52:14.282940 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.282629 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/553c8b63-e975-498f-9313-878fbf99066d-workload-socket" pod "553c8b63-e975-498f-9313-878fbf99066d" (UID: "553c8b63-e975-498f-9313-878fbf99066d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:52:14.282940 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.282676 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-workload-certs" pod "d75821f5-34cd-4eeb-a9cc-a0c19573f0f8" (UID: "d75821f5-34cd-4eeb-a9cc-a0c19573f0f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:52:14.282940 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.282714 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-istio-data" pod "d75821f5-34cd-4eeb-a9cc-a0c19573f0f8" (UID: "d75821f5-34cd-4eeb-a9cc-a0c19573f0f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:52:14.283211 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.283178 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/553c8b63-e975-498f-9313-878fbf99066d-credential-socket" pod "553c8b63-e975-498f-9313-878fbf99066d" (UID: "553c8b63-e975-498f-9313-878fbf99066d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:52:14.283904 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.283874 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-istio-podinfo" pod "d75821f5-34cd-4eeb-a9cc-a0c19573f0f8" (UID: "d75821f5-34cd-4eeb-a9cc-a0c19573f0f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:52:14.284484 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.284454 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-istio-envoy" pod "d75821f5-34cd-4eeb-a9cc-a0c19573f0f8" (UID: "d75821f5-34cd-4eeb-a9cc-a0c19573f0f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:52:14.285003 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.284979 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-istio-token" pod "d75821f5-34cd-4eeb-a9cc-a0c19573f0f8" (UID: "d75821f5-34cd-4eeb-a9cc-a0c19573f0f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:52:14.285211 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.285190 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/553c8b63-e975-498f-9313-878fbf99066d-istio-podinfo" pod "553c8b63-e975-498f-9313-878fbf99066d" (UID: "553c8b63-e975-498f-9313-878fbf99066d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:52:14.285291 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.285204 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-kube-api-access-9kgzh" pod "d75821f5-34cd-4eeb-a9cc-a0c19573f0f8" (UID: "d75821f5-34cd-4eeb-a9cc-a0c19573f0f8"). InnerVolumeSpecName "kube-api-access-9kgzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:52:14.285688 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.285670 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/553c8b63-e975-498f-9313-878fbf99066d-istio-token" pod "553c8b63-e975-498f-9313-878fbf99066d" (UID: "553c8b63-e975-498f-9313-878fbf99066d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:52:14.285924 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.285904 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/553c8b63-e975-498f-9313-878fbf99066d-kube-api-access-spvwr" pod "553c8b63-e975-498f-9313-878fbf99066d" (UID: "553c8b63-e975-498f-9313-878fbf99066d"). InnerVolumeSpecName "kube-api-access-spvwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:52:14.286535 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.286507 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/553c8b63-e975-498f-9313-878fbf99066d-istio-envoy" pod "553c8b63-e975-498f-9313-878fbf99066d" (UID: "553c8b63-e975-498f-9313-878fbf99066d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:52:14.383906 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.383867 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/553c8b63-e975-498f-9313-878fbf99066d-workload-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:52:14.383906 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.383898 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-istio-envoy\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:52:14.383906 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.383908 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-istio-podinfo\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:52:14.383906 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.383917 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-workload-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:52:14.384223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.383925 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/553c8b63-e975-498f-9313-878fbf99066d-istio-envoy\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:52:14.384223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.383934 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kgzh\" (UniqueName: \"kubernetes.io/projected/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-kube-api-access-9kgzh\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:52:14.384223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.383944 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spvwr\" (UniqueName: \"kubernetes.io/projected/553c8b63-e975-498f-9313-878fbf99066d-kube-api-access-spvwr\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:52:14.384223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.383952 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/553c8b63-e975-498f-9313-878fbf99066d-istio-token\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:52:14.384223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.383960 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-istio-token\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:52:14.384223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.383968 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/553c8b63-e975-498f-9313-878fbf99066d-credential-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:52:14.384223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.383978 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/553c8b63-e975-498f-9313-878fbf99066d-istio-podinfo\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:52:14.384223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.383986 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/553c8b63-e975-498f-9313-878fbf99066d-workload-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:52:14.384223 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.383994 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8-istio-data\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:52:14.894950 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.894912 4952 generic.go:356] "Generic (PLEG): container finished" podID="d75821f5-34cd-4eeb-a9cc-a0c19573f0f8" containerID="cff7a5c11e1ce818e4db267beceec569eb4bbe27eac34c76ce518b1b3eee9ae3" exitCode=0 Oct 30 23:52:14.895446 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.895075 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" Oct 30 23:52:14.895446 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.895086 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" event={"ID":"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8","Type":"ContainerDied","Data":"cff7a5c11e1ce818e4db267beceec569eb4bbe27eac34c76ce518b1b3eee9ae3"} Oct 30 23:52:14.895446 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.895151 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4" event={"ID":"d75821f5-34cd-4eeb-a9cc-a0c19573f0f8","Type":"ContainerDied","Data":"a7a9e71b34154b4f92af06d6d09dc9152a6cb0b3b3559e747791427932a1aaa1"} Oct 30 23:52:14.895446 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.895179 4952 scope.go:122] "RemoveContainer" containerID="cff7a5c11e1ce818e4db267beceec569eb4bbe27eac34c76ce518b1b3eee9ae3" Oct 30 23:52:14.897373 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.897345 4952 generic.go:356] "Generic (PLEG): container finished" podID="553c8b63-e975-498f-9313-878fbf99066d" containerID="0e539ca1c5acab406e0fcffffeb55994cadd84ab227f590dadf31a310a3c588a" exitCode=0 Oct 30 23:52:14.897507 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.897418 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" Oct 30 23:52:14.897507 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.897439 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" event={"ID":"553c8b63-e975-498f-9313-878fbf99066d","Type":"ContainerDied","Data":"0e539ca1c5acab406e0fcffffeb55994cadd84ab227f590dadf31a310a3c588a"} Oct 30 23:52:14.897507 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.897483 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp" event={"ID":"553c8b63-e975-498f-9313-878fbf99066d","Type":"ContainerDied","Data":"fa2c4d7f1f57261dc7f76d04e662b19ccecb9b7464c0822eb60234a443c61d3e"} Oct 30 23:52:14.907848 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.907822 4952 scope.go:122] "RemoveContainer" containerID="cff7a5c11e1ce818e4db267beceec569eb4bbe27eac34c76ce518b1b3eee9ae3" Oct 30 23:52:14.908378 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:52:14.908354 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cff7a5c11e1ce818e4db267beceec569eb4bbe27eac34c76ce518b1b3eee9ae3\": container with ID starting with cff7a5c11e1ce818e4db267beceec569eb4bbe27eac34c76ce518b1b3eee9ae3 not found: ID does not exist" containerID="cff7a5c11e1ce818e4db267beceec569eb4bbe27eac34c76ce518b1b3eee9ae3" Oct 30 23:52:14.908474 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.908385 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cff7a5c11e1ce818e4db267beceec569eb4bbe27eac34c76ce518b1b3eee9ae3"} err="failed to get container status \"cff7a5c11e1ce818e4db267beceec569eb4bbe27eac34c76ce518b1b3eee9ae3\": rpc error: code = NotFound desc = could not find container \"cff7a5c11e1ce818e4db267beceec569eb4bbe27eac34c76ce518b1b3eee9ae3\": container with ID starting with cff7a5c11e1ce818e4db267beceec569eb4bbe27eac34c76ce518b1b3eee9ae3 not found: ID does not exist" Oct 30 23:52:14.908474 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.908407 4952 scope.go:122] "RemoveContainer" containerID="0e539ca1c5acab406e0fcffffeb55994cadd84ab227f590dadf31a310a3c588a" Oct 30 23:52:14.919675 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.919656 4952 scope.go:122] "RemoveContainer" containerID="0e539ca1c5acab406e0fcffffeb55994cadd84ab227f590dadf31a310a3c588a" Oct 30 23:52:14.919988 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:52:14.919970 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e539ca1c5acab406e0fcffffeb55994cadd84ab227f590dadf31a310a3c588a\": container with ID starting with 0e539ca1c5acab406e0fcffffeb55994cadd84ab227f590dadf31a310a3c588a not found: ID does not exist" containerID="0e539ca1c5acab406e0fcffffeb55994cadd84ab227f590dadf31a310a3c588a" Oct 30 23:52:14.920080 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:14.920003 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e539ca1c5acab406e0fcffffeb55994cadd84ab227f590dadf31a310a3c588a"} err="failed to get container status \"0e539ca1c5acab406e0fcffffeb55994cadd84ab227f590dadf31a310a3c588a\": rpc error: code = NotFound desc = could not find container \"0e539ca1c5acab406e0fcffffeb55994cadd84ab227f590dadf31a310a3c588a\": container with ID starting with 0e539ca1c5acab406e0fcffffeb55994cadd84ab227f590dadf31a310a3c588a not found: ID does not exist" Oct 30 23:52:15.063521 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:15.063486 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4"] Oct 30 23:52:15.070896 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:15.070857 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-5vlq4"] Oct 30 23:52:15.187350 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:15.187251 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp"] Oct 30 23:52:15.196247 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:15.196214 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-mclbp"] Oct 30 23:52:16.188543 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:16.188505 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9"] Oct 30 23:52:16.188993 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:16.188744 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9" podUID="305e64da-56e7-4fbb-b9fa-a7f486ed1dac" containerName="discovery" containerID="cri-o://c6479aece7c37b72c59d5cb18e908a14dd685036306922b4892afc31bb8e0a35" gracePeriod=30 Oct 30 23:52:16.473901 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:16.473878 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9" Oct 30 23:52:16.516442 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:16.516408 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-istio-csr-ca-configmap\") pod \"305e64da-56e7-4fbb-b9fa-a7f486ed1dac\" (UID: \"305e64da-56e7-4fbb-b9fa-a7f486ed1dac\") " Oct 30 23:52:16.516638 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:16.516492 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-kube-api-access-j2lms\" (UniqueName: \"kubernetes.io/projected/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-kube-api-access-j2lms\") pod \"305e64da-56e7-4fbb-b9fa-a7f486ed1dac\" (UID: \"305e64da-56e7-4fbb-b9fa-a7f486ed1dac\") " Oct 30 23:52:16.516638 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:16.516536 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-istio-csr-dns-cert\") pod \"305e64da-56e7-4fbb-b9fa-a7f486ed1dac\" (UID: \"305e64da-56e7-4fbb-b9fa-a7f486ed1dac\") " Oct 30 23:52:16.516752 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:16.516728 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-istio-token\" (UniqueName: \"kubernetes.io/projected/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-istio-token\") pod \"305e64da-56e7-4fbb-b9fa-a7f486ed1dac\" (UID: \"305e64da-56e7-4fbb-b9fa-a7f486ed1dac\") " Oct 30 23:52:16.516815 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:16.516775 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-istio-kubeconfig\") pod \"305e64da-56e7-4fbb-b9fa-a7f486ed1dac\" (UID: \"305e64da-56e7-4fbb-b9fa-a7f486ed1dac\") " Oct 30 23:52:16.516873 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:16.516830 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-cacerts\" (UniqueName: \"kubernetes.io/secret/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-cacerts\") pod \"305e64da-56e7-4fbb-b9fa-a7f486ed1dac\" (UID: \"305e64da-56e7-4fbb-b9fa-a7f486ed1dac\") " Oct 30 23:52:16.516934 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:16.516877 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-local-certs\") pod \"305e64da-56e7-4fbb-b9fa-a7f486ed1dac\" (UID: \"305e64da-56e7-4fbb-b9fa-a7f486ed1dac\") " Oct 30 23:52:16.516934 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:16.516886 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-istio-csr-ca-configmap" pod "305e64da-56e7-4fbb-b9fa-a7f486ed1dac" (UID: "305e64da-56e7-4fbb-b9fa-a7f486ed1dac"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:52:16.517374 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:16.517311 4952 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-istio-csr-ca-configmap\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:52:16.519681 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:16.519641 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-kube-api-access-j2lms" pod "305e64da-56e7-4fbb-b9fa-a7f486ed1dac" (UID: "305e64da-56e7-4fbb-b9fa-a7f486ed1dac"). InnerVolumeSpecName "kube-api-access-j2lms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:52:16.519950 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:16.519926 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-cacerts" pod "305e64da-56e7-4fbb-b9fa-a7f486ed1dac" (UID: "305e64da-56e7-4fbb-b9fa-a7f486ed1dac"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:52:16.520193 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:16.520167 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-local-certs" pod "305e64da-56e7-4fbb-b9fa-a7f486ed1dac" (UID: "305e64da-56e7-4fbb-b9fa-a7f486ed1dac"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:52:16.520474 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:16.520452 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-istio-kubeconfig" pod "305e64da-56e7-4fbb-b9fa-a7f486ed1dac" (UID: "305e64da-56e7-4fbb-b9fa-a7f486ed1dac"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:52:16.520537 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:16.520487 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-istio-csr-dns-cert" pod "305e64da-56e7-4fbb-b9fa-a7f486ed1dac" (UID: "305e64da-56e7-4fbb-b9fa-a7f486ed1dac"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:52:16.520577 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:16.520564 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-istio-token" pod "305e64da-56e7-4fbb-b9fa-a7f486ed1dac" (UID: "305e64da-56e7-4fbb-b9fa-a7f486ed1dac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:52:16.618877 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:16.618842 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-istio-token\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:52:16.618877 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:16.618874 4952 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-istio-kubeconfig\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:52:16.619096 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:16.618885 4952 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-cacerts\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:52:16.619096 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:16.618895 4952 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-local-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:52:16.619096 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:16.618925 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2lms\" (UniqueName: \"kubernetes.io/projected/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-kube-api-access-j2lms\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:52:16.619096 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:16.618936 4952 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/305e64da-56e7-4fbb-b9fa-a7f486ed1dac-istio-csr-dns-cert\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:52:16.915090 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:16.915055 4952 generic.go:356] "Generic (PLEG): container finished" podID="305e64da-56e7-4fbb-b9fa-a7f486ed1dac" containerID="c6479aece7c37b72c59d5cb18e908a14dd685036306922b4892afc31bb8e0a35" exitCode=0 Oct 30 23:52:16.915262 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:16.915140 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9" Oct 30 23:52:16.915262 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:16.915138 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9" event={"ID":"305e64da-56e7-4fbb-b9fa-a7f486ed1dac","Type":"ContainerDied","Data":"c6479aece7c37b72c59d5cb18e908a14dd685036306922b4892afc31bb8e0a35"} Oct 30 23:52:16.915262 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:16.915225 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9" event={"ID":"305e64da-56e7-4fbb-b9fa-a7f486ed1dac","Type":"ContainerDied","Data":"91f342561b62ca8a7f32085baf617fb326d752d05613c39b711389456d53c104"} Oct 30 23:52:16.915262 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:16.915248 4952 scope.go:122] "RemoveContainer" containerID="c6479aece7c37b72c59d5cb18e908a14dd685036306922b4892afc31bb8e0a35" Oct 30 23:52:16.926967 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:16.926946 4952 scope.go:122] "RemoveContainer" containerID="c6479aece7c37b72c59d5cb18e908a14dd685036306922b4892afc31bb8e0a35" Oct 30 23:52:16.927277 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:52:16.927256 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6479aece7c37b72c59d5cb18e908a14dd685036306922b4892afc31bb8e0a35\": container with ID starting with c6479aece7c37b72c59d5cb18e908a14dd685036306922b4892afc31bb8e0a35 not found: ID does not exist" containerID="c6479aece7c37b72c59d5cb18e908a14dd685036306922b4892afc31bb8e0a35" Oct 30 23:52:16.927326 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:16.927287 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6479aece7c37b72c59d5cb18e908a14dd685036306922b4892afc31bb8e0a35"} err="failed to get container status \"c6479aece7c37b72c59d5cb18e908a14dd685036306922b4892afc31bb8e0a35\": rpc error: code = NotFound desc = could not find container \"c6479aece7c37b72c59d5cb18e908a14dd685036306922b4892afc31bb8e0a35\": container with ID starting with c6479aece7c37b72c59d5cb18e908a14dd685036306922b4892afc31bb8e0a35 not found: ID does not exist" Oct 30 23:52:16.971391 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:16.971366 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="553c8b63-e975-498f-9313-878fbf99066d" path="/var/lib/kubelet/pods/553c8b63-e975-498f-9313-878fbf99066d/volumes" Oct 30 23:52:16.971864 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:16.971842 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d75821f5-34cd-4eeb-a9cc-a0c19573f0f8" path="/var/lib/kubelet/pods/d75821f5-34cd-4eeb-a9cc-a0c19573f0f8/volumes" Oct 30 23:52:17.082923 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:17.082891 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9"] Oct 30 23:52:17.132139 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:17.132104 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t42s9"] Oct 30 23:52:18.970435 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:18.970399 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="305e64da-56e7-4fbb-b9fa-a7f486ed1dac" path="/var/lib/kubelet/pods/305e64da-56e7-4fbb-b9fa-a7f486ed1dac/volumes" Oct 30 23:52:19.156092 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:19.156050 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:21.142184 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:52:21.142137 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:52:23.963310 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:23.963276 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:26.222010 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:26.221972 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:26.446456 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:26.446418 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:28.723335 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:28.723298 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:33.566757 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:33.566714 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:38.363053 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:38.362991 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:43.169018 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:43.168964 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:47.982492 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:47.982443 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:51.142229 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:52:51.142192 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:52:52.786621 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:52.786593 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:52:57.671182 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:52:57.671149 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:02.469971 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:02.469920 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:07.272049 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:07.271999 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:07.566137 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:07.566052 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc"] Oct 30 23:53:07.566345 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:07.566321 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" podUID="414e5e6c-0f02-40f5-806f-0ad048215d6a" containerName="istio-proxy" containerID="cri-o://4b24d984ef4a14d826ad729bd5e747c1e60a60997ed9f183eed3714200becf3d" gracePeriod=30 Oct 30 23:53:12.067310 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:12.067273 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:12.849463 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:12.849434 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" Oct 30 23:53:12.915596 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:12.915570 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/414e5e6c-0f02-40f5-806f-0ad048215d6a-kube-api-access-rn8wf\" (UniqueName: \"kubernetes.io/projected/414e5e6c-0f02-40f5-806f-0ad048215d6a-kube-api-access-rn8wf\") pod \"414e5e6c-0f02-40f5-806f-0ad048215d6a\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " Oct 30 23:53:12.915773 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:12.915604 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/414e5e6c-0f02-40f5-806f-0ad048215d6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/414e5e6c-0f02-40f5-806f-0ad048215d6a-istio-data\") pod \"414e5e6c-0f02-40f5-806f-0ad048215d6a\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " Oct 30 23:53:12.915773 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:12.915624 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/414e5e6c-0f02-40f5-806f-0ad048215d6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/414e5e6c-0f02-40f5-806f-0ad048215d6a-workload-socket\") pod \"414e5e6c-0f02-40f5-806f-0ad048215d6a\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " Oct 30 23:53:12.915773 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:12.915645 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/414e5e6c-0f02-40f5-806f-0ad048215d6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/414e5e6c-0f02-40f5-806f-0ad048215d6a-workload-certs\") pod \"414e5e6c-0f02-40f5-806f-0ad048215d6a\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " Oct 30 23:53:12.915773 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:12.915702 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/414e5e6c-0f02-40f5-806f-0ad048215d6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/414e5e6c-0f02-40f5-806f-0ad048215d6a-istio-podinfo\") pod \"414e5e6c-0f02-40f5-806f-0ad048215d6a\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " Oct 30 23:53:12.915773 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:12.915735 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/414e5e6c-0f02-40f5-806f-0ad048215d6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/414e5e6c-0f02-40f5-806f-0ad048215d6a-credential-socket\") pod \"414e5e6c-0f02-40f5-806f-0ad048215d6a\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " Oct 30 23:53:12.916084 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:12.915823 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/414e5e6c-0f02-40f5-806f-0ad048215d6a-istio-token\" (UniqueName: \"kubernetes.io/projected/414e5e6c-0f02-40f5-806f-0ad048215d6a-istio-token\") pod \"414e5e6c-0f02-40f5-806f-0ad048215d6a\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " Oct 30 23:53:12.916084 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:12.915852 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/414e5e6c-0f02-40f5-806f-0ad048215d6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/414e5e6c-0f02-40f5-806f-0ad048215d6a-istiod-ca-cert\") pod \"414e5e6c-0f02-40f5-806f-0ad048215d6a\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " Oct 30 23:53:12.916084 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:12.915889 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/414e5e6c-0f02-40f5-806f-0ad048215d6a-istio-data" pod "414e5e6c-0f02-40f5-806f-0ad048215d6a" (UID: "414e5e6c-0f02-40f5-806f-0ad048215d6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:53:12.916084 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:12.915976 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/414e5e6c-0f02-40f5-806f-0ad048215d6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/414e5e6c-0f02-40f5-806f-0ad048215d6a-istio-envoy\") pod \"414e5e6c-0f02-40f5-806f-0ad048215d6a\" (UID: \"414e5e6c-0f02-40f5-806f-0ad048215d6a\") " Oct 30 23:53:12.916084 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:12.916018 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/414e5e6c-0f02-40f5-806f-0ad048215d6a-workload-certs" pod "414e5e6c-0f02-40f5-806f-0ad048215d6a" (UID: "414e5e6c-0f02-40f5-806f-0ad048215d6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:53:12.916379 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:12.916029 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/414e5e6c-0f02-40f5-806f-0ad048215d6a-workload-socket" pod "414e5e6c-0f02-40f5-806f-0ad048215d6a" (UID: "414e5e6c-0f02-40f5-806f-0ad048215d6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:53:12.916379 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:12.916365 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/414e5e6c-0f02-40f5-806f-0ad048215d6a-istiod-ca-cert" pod "414e5e6c-0f02-40f5-806f-0ad048215d6a" (UID: "414e5e6c-0f02-40f5-806f-0ad048215d6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:53:12.916500 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:12.916465 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/414e5e6c-0f02-40f5-806f-0ad048215d6a-credential-socket" pod "414e5e6c-0f02-40f5-806f-0ad048215d6a" (UID: "414e5e6c-0f02-40f5-806f-0ad048215d6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:53:12.916563 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:12.916529 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/414e5e6c-0f02-40f5-806f-0ad048215d6a-istio-data\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:53:12.916563 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:12.916548 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/414e5e6c-0f02-40f5-806f-0ad048215d6a-workload-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:53:12.916563 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:12.916562 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/414e5e6c-0f02-40f5-806f-0ad048215d6a-workload-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:53:12.916563 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:12.916575 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/414e5e6c-0f02-40f5-806f-0ad048215d6a-istiod-ca-cert\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:53:12.918646 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:12.918622 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/414e5e6c-0f02-40f5-806f-0ad048215d6a-istio-podinfo" pod "414e5e6c-0f02-40f5-806f-0ad048215d6a" (UID: "414e5e6c-0f02-40f5-806f-0ad048215d6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:53:12.918911 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:12.918882 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/414e5e6c-0f02-40f5-806f-0ad048215d6a-istio-token" pod "414e5e6c-0f02-40f5-806f-0ad048215d6a" (UID: "414e5e6c-0f02-40f5-806f-0ad048215d6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:53:12.919029 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:12.918937 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/414e5e6c-0f02-40f5-806f-0ad048215d6a-istio-envoy" pod "414e5e6c-0f02-40f5-806f-0ad048215d6a" (UID: "414e5e6c-0f02-40f5-806f-0ad048215d6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:53:12.919288 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:12.919262 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/414e5e6c-0f02-40f5-806f-0ad048215d6a-kube-api-access-rn8wf" pod "414e5e6c-0f02-40f5-806f-0ad048215d6a" (UID: "414e5e6c-0f02-40f5-806f-0ad048215d6a"). InnerVolumeSpecName "kube-api-access-rn8wf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:53:12.990419 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:12.990394 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:53:13.017624 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:13.017598 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/414e5e6c-0f02-40f5-806f-0ad048215d6a-istio-token\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:53:13.017820 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:13.017795 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/414e5e6c-0f02-40f5-806f-0ad048215d6a-istio-envoy\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:53:13.017933 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:13.017821 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn8wf\" (UniqueName: \"kubernetes.io/projected/414e5e6c-0f02-40f5-806f-0ad048215d6a-kube-api-access-rn8wf\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:53:13.017933 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:13.017836 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/414e5e6c-0f02-40f5-806f-0ad048215d6a-istio-podinfo\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:53:13.017933 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:13.017849 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/414e5e6c-0f02-40f5-806f-0ad048215d6a-credential-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:53:13.287914 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:13.287814 4952 generic.go:356] "Generic (PLEG): container finished" podID="414e5e6c-0f02-40f5-806f-0ad048215d6a" containerID="4b24d984ef4a14d826ad729bd5e747c1e60a60997ed9f183eed3714200becf3d" exitCode=0 Oct 30 23:53:13.288944 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:13.287947 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" Oct 30 23:53:13.288944 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:13.288014 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" event={"ID":"414e5e6c-0f02-40f5-806f-0ad048215d6a","Type":"ContainerDied","Data":"4b24d984ef4a14d826ad729bd5e747c1e60a60997ed9f183eed3714200becf3d"} Oct 30 23:53:13.288944 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:13.288113 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc" event={"ID":"414e5e6c-0f02-40f5-806f-0ad048215d6a","Type":"ContainerDied","Data":"d86e976cd190c2fe4e091ba72006fdf059ea9126e7c5e3b76ea4b1bd3d2541c6"} Oct 30 23:53:13.288944 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:13.288145 4952 scope.go:122] "RemoveContainer" containerID="4b24d984ef4a14d826ad729bd5e747c1e60a60997ed9f183eed3714200becf3d" Oct 30 23:53:13.300822 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:13.300795 4952 scope.go:122] "RemoveContainer" containerID="4b24d984ef4a14d826ad729bd5e747c1e60a60997ed9f183eed3714200becf3d" Oct 30 23:53:13.301232 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:53:13.301178 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b24d984ef4a14d826ad729bd5e747c1e60a60997ed9f183eed3714200becf3d\": container with ID starting with 4b24d984ef4a14d826ad729bd5e747c1e60a60997ed9f183eed3714200becf3d not found: ID does not exist" containerID="4b24d984ef4a14d826ad729bd5e747c1e60a60997ed9f183eed3714200becf3d" Oct 30 23:53:13.301232 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:13.301216 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b24d984ef4a14d826ad729bd5e747c1e60a60997ed9f183eed3714200becf3d"} err="failed to get container status \"4b24d984ef4a14d826ad729bd5e747c1e60a60997ed9f183eed3714200becf3d\": rpc error: code = NotFound desc = could not find container \"4b24d984ef4a14d826ad729bd5e747c1e60a60997ed9f183eed3714200becf3d\": container with ID starting with 4b24d984ef4a14d826ad729bd5e747c1e60a60997ed9f183eed3714200becf3d not found: ID does not exist" Oct 30 23:53:13.453568 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:13.453535 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc"] Oct 30 23:53:13.502854 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:13.502824 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vgtcc"] Oct 30 23:53:14.970320 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:14.970286 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="414e5e6c-0f02-40f5-806f-0ad048215d6a" path="/var/lib/kubelet/pods/414e5e6c-0f02-40f5-806f-0ad048215d6a/volumes" Oct 30 23:53:16.261867 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:16.261835 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:17.844439 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:17.844398 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:18.087068 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:18.087014 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:18.334357 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:18.334273 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:18.877211 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:18.877158 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:21.142482 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:53:21.142448 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:53:23.150124 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:23.150094 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:24.969376 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:24.969336 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:25.211598 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:25.211564 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:25.578879 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:25.578845 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:25.926285 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:25.926250 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:26.561782 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:26.561748 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:28.311923 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:28.311884 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:30.820283 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:30.820205 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d"] Oct 30 23:53:31.043500 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:31.043450 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:33.311128 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:33.311094 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:33.530386 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:33.530353 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:37.327864 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:37.327825 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:37.548182 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:37.548147 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:38.158530 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:38.158492 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:43.070313 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:43.070273 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:45.841142 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:45.841103 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" podUID="da4935dd-c148-4cc4-84ab-1bfd91cce9b4" containerName="main" containerID="cri-o://b8523a64b94bc7aec9fed2c260c618b394a0c4217df92c65adc3da67b70f94a9" gracePeriod=45 Oct 30 23:53:46.110014 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:46.109962 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" Oct 30 23:53:46.254581 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:46.254552 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-dshm\") pod \"da4935dd-c148-4cc4-84ab-1bfd91cce9b4\" (UID: \"da4935dd-c148-4cc4-84ab-1bfd91cce9b4\") " Oct 30 23:53:46.254758 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:46.254597 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-model-cache\") pod \"da4935dd-c148-4cc4-84ab-1bfd91cce9b4\" (UID: \"da4935dd-c148-4cc4-84ab-1bfd91cce9b4\") " Oct 30 23:53:46.254758 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:46.254629 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-tls-certs\" (UniqueName: \"kubernetes.io/secret/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-tls-certs\") pod \"da4935dd-c148-4cc4-84ab-1bfd91cce9b4\" (UID: \"da4935dd-c148-4cc4-84ab-1bfd91cce9b4\") " Oct 30 23:53:46.254758 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:46.254715 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-tmp-dir\") pod \"da4935dd-c148-4cc4-84ab-1bfd91cce9b4\" (UID: \"da4935dd-c148-4cc4-84ab-1bfd91cce9b4\") " Oct 30 23:53:46.254945 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:46.254767 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-home\" (UniqueName: \"kubernetes.io/empty-dir/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-home\") pod \"da4935dd-c148-4cc4-84ab-1bfd91cce9b4\" (UID: \"da4935dd-c148-4cc4-84ab-1bfd91cce9b4\") " Oct 30 23:53:46.254945 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:46.254794 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-kube-api-access-fgdl2\" (UniqueName: \"kubernetes.io/projected/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-kube-api-access-fgdl2\") pod \"da4935dd-c148-4cc4-84ab-1bfd91cce9b4\" (UID: \"da4935dd-c148-4cc4-84ab-1bfd91cce9b4\") " Oct 30 23:53:46.254945 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:46.254903 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-model-cache" pod "da4935dd-c148-4cc4-84ab-1bfd91cce9b4" (UID: "da4935dd-c148-4cc4-84ab-1bfd91cce9b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:53:46.255144 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:46.254967 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-tmp-dir" pod "da4935dd-c148-4cc4-84ab-1bfd91cce9b4" (UID: "da4935dd-c148-4cc4-84ab-1bfd91cce9b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:53:46.255144 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:46.255080 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-home" pod "da4935dd-c148-4cc4-84ab-1bfd91cce9b4" (UID: "da4935dd-c148-4cc4-84ab-1bfd91cce9b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:53:46.255330 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:46.255271 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-tmp-dir\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:53:46.255330 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:46.255293 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-home\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:53:46.255330 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:46.255309 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-model-cache\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:53:46.257500 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:46.257472 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-dshm" pod "da4935dd-c148-4cc4-84ab-1bfd91cce9b4" (UID: "da4935dd-c148-4cc4-84ab-1bfd91cce9b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:53:46.258070 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:46.258001 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-tls-certs" pod "da4935dd-c148-4cc4-84ab-1bfd91cce9b4" (UID: "da4935dd-c148-4cc4-84ab-1bfd91cce9b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:53:46.258951 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:46.258922 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-kube-api-access-fgdl2" pod "da4935dd-c148-4cc4-84ab-1bfd91cce9b4" (UID: "da4935dd-c148-4cc4-84ab-1bfd91cce9b4"). InnerVolumeSpecName "kube-api-access-fgdl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:53:46.356497 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:46.356467 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgdl2\" (UniqueName: \"kubernetes.io/projected/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-kube-api-access-fgdl2\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:53:46.356497 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:46.356494 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-dshm\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:53:46.356497 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:46.356505 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4935dd-c148-4cc4-84ab-1bfd91cce9b4-tls-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:53:46.498100 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:46.498001 4952 generic.go:356] "Generic (PLEG): container finished" podID="da4935dd-c148-4cc4-84ab-1bfd91cce9b4" containerID="b8523a64b94bc7aec9fed2c260c618b394a0c4217df92c65adc3da67b70f94a9" exitCode=0 Oct 30 23:53:46.498100 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:46.498060 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" Oct 30 23:53:46.498100 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:46.498078 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" event={"ID":"da4935dd-c148-4cc4-84ab-1bfd91cce9b4","Type":"ContainerDied","Data":"b8523a64b94bc7aec9fed2c260c618b394a0c4217df92c65adc3da67b70f94a9"} Oct 30 23:53:46.498314 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:46.498126 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d" event={"ID":"da4935dd-c148-4cc4-84ab-1bfd91cce9b4","Type":"ContainerDied","Data":"5db4c0780e9e152107889799c3363a455d4c0d06e53b4060fe7e291b1a6c834a"} Oct 30 23:53:46.498314 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:46.498151 4952 scope.go:122] "RemoveContainer" containerID="b8523a64b94bc7aec9fed2c260c618b394a0c4217df92c65adc3da67b70f94a9" Oct 30 23:53:46.516987 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:46.516963 4952 scope.go:122] "RemoveContainer" containerID="b8523a64b94bc7aec9fed2c260c618b394a0c4217df92c65adc3da67b70f94a9" Oct 30 23:53:46.517319 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:53:46.517299 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8523a64b94bc7aec9fed2c260c618b394a0c4217df92c65adc3da67b70f94a9\": container with ID starting with b8523a64b94bc7aec9fed2c260c618b394a0c4217df92c65adc3da67b70f94a9 not found: ID does not exist" containerID="b8523a64b94bc7aec9fed2c260c618b394a0c4217df92c65adc3da67b70f94a9" Oct 30 23:53:46.517418 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:46.517328 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8523a64b94bc7aec9fed2c260c618b394a0c4217df92c65adc3da67b70f94a9"} err="failed to get container status \"b8523a64b94bc7aec9fed2c260c618b394a0c4217df92c65adc3da67b70f94a9\": rpc error: code = NotFound desc = could not find container \"b8523a64b94bc7aec9fed2c260c618b394a0c4217df92c65adc3da67b70f94a9\": container with ID starting with b8523a64b94bc7aec9fed2c260c618b394a0c4217df92c65adc3da67b70f94a9 not found: ID does not exist" Oct 30 23:53:46.662873 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:46.662847 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d"] Oct 30 23:53:46.712426 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:46.712402 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2cfb80/simulator-w3-2cfb80-kserve-7cf744bb95-frh9d"] Oct 30 23:53:46.970132 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:46.970106 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da4935dd-c148-4cc4-84ab-1bfd91cce9b4" path="/var/lib/kubelet/pods/da4935dd-c148-4cc4-84ab-1bfd91cce9b4/volumes" Oct 30 23:53:47.872863 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:47.872830 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:51.142426 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:53:51.142394 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:53:52.041447 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:52.041412 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:53:52.673281 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:52.673236 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:52.898153 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:52.898110 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:53:57.467593 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:53:57.467547 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:02.580439 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:54:02.580407 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:07.366570 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:54:07.366535 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:07.683366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:54:07.683271 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:07.910067 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:54:07.909995 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:08.900453 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:54:08.900419 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:12.167871 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:54:12.167842 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:14.589564 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:54:14.589532 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:16.984082 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:54:16.984012 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:21.142006 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:54:21.141969 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:54:22.869569 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:54:22.869536 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:23.329276 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:54:23.329203 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:23.553736 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:54:23.553702 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:26.623715 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:54:26.623685 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:26.912359 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:54:26.912275 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:31.531777 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:54:31.531717 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:36.651313 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:54:36.651275 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:37.834054 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:54:37.834006 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:41.756384 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:54:41.756352 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:46.532300 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:54:46.532259 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:47.099326 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:54:47.099290 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq"] Oct 30 23:54:47.263604 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:54:47.263566 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:48.365536 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:54:48.365495 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:48.638429 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:54:48.638351 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:50.477941 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:54:50.477901 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:50.728241 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:54:50.728160 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:51.141708 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:54:51.141658 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:54:51.642846 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:54:51.642815 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:52.336921 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:54:52.336889 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:54:54.942389 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:54:54.942352 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:55.186961 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:54:55.186913 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:56.789317 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:54:56.789279 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:01.719069 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:01.718951 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:02.121137 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:02.121069 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq" podUID="39ddf73a-4de1-404a-8a9a-276804aa9209" containerName="main" containerID="cri-o://d4eec5bb1d2355560ecfcf0977b4615951cb6fbdbf7c05b784c4597b0c69ab12" gracePeriod=45 Oct 30 23:55:02.388063 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:02.387986 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq" Oct 30 23:55:02.511738 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:02.511704 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:02.555650 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:02.555622 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ddf73a-4de1-404a-8a9a-276804aa9209-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ddf73a-4de1-404a-8a9a-276804aa9209-model-cache\") pod \"39ddf73a-4de1-404a-8a9a-276804aa9209\" (UID: \"39ddf73a-4de1-404a-8a9a-276804aa9209\") " Oct 30 23:55:02.555801 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:02.555659 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39ddf73a-4de1-404a-8a9a-276804aa9209-kube-api-access-sn6sg\" (UniqueName: \"kubernetes.io/projected/39ddf73a-4de1-404a-8a9a-276804aa9209-kube-api-access-sn6sg\") pod \"39ddf73a-4de1-404a-8a9a-276804aa9209\" (UID: \"39ddf73a-4de1-404a-8a9a-276804aa9209\") " Oct 30 23:55:02.555801 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:02.555686 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ddf73a-4de1-404a-8a9a-276804aa9209-home\" (UniqueName: \"kubernetes.io/empty-dir/39ddf73a-4de1-404a-8a9a-276804aa9209-home\") pod \"39ddf73a-4de1-404a-8a9a-276804aa9209\" (UID: \"39ddf73a-4de1-404a-8a9a-276804aa9209\") " Oct 30 23:55:02.555801 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:02.555734 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ddf73a-4de1-404a-8a9a-276804aa9209-dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ddf73a-4de1-404a-8a9a-276804aa9209-dshm\") pod \"39ddf73a-4de1-404a-8a9a-276804aa9209\" (UID: \"39ddf73a-4de1-404a-8a9a-276804aa9209\") " Oct 30 23:55:02.556022 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:02.555814 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ddf73a-4de1-404a-8a9a-276804aa9209-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ddf73a-4de1-404a-8a9a-276804aa9209-tmp-dir\") pod \"39ddf73a-4de1-404a-8a9a-276804aa9209\" (UID: \"39ddf73a-4de1-404a-8a9a-276804aa9209\") " Oct 30 23:55:02.556022 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:02.555861 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39ddf73a-4de1-404a-8a9a-276804aa9209-tls-certs\" (UniqueName: \"kubernetes.io/secret/39ddf73a-4de1-404a-8a9a-276804aa9209-tls-certs\") pod \"39ddf73a-4de1-404a-8a9a-276804aa9209\" (UID: \"39ddf73a-4de1-404a-8a9a-276804aa9209\") " Oct 30 23:55:02.556022 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:02.555957 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ddf73a-4de1-404a-8a9a-276804aa9209-model-cache" pod "39ddf73a-4de1-404a-8a9a-276804aa9209" (UID: "39ddf73a-4de1-404a-8a9a-276804aa9209"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:55:02.556022 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:02.555965 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ddf73a-4de1-404a-8a9a-276804aa9209-home" pod "39ddf73a-4de1-404a-8a9a-276804aa9209" (UID: "39ddf73a-4de1-404a-8a9a-276804aa9209"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:55:02.556321 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:02.556233 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ddf73a-4de1-404a-8a9a-276804aa9209-model-cache\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:55:02.556321 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:02.556248 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39ddf73a-4de1-404a-8a9a-276804aa9209-home\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:55:02.556485 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:02.556461 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ddf73a-4de1-404a-8a9a-276804aa9209-tmp-dir" pod "39ddf73a-4de1-404a-8a9a-276804aa9209" (UID: "39ddf73a-4de1-404a-8a9a-276804aa9209"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:55:02.558179 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:02.558156 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39ddf73a-4de1-404a-8a9a-276804aa9209-tls-certs" pod "39ddf73a-4de1-404a-8a9a-276804aa9209" (UID: "39ddf73a-4de1-404a-8a9a-276804aa9209"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:55:02.558278 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:02.558249 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ddf73a-4de1-404a-8a9a-276804aa9209-kube-api-access-sn6sg" pod "39ddf73a-4de1-404a-8a9a-276804aa9209" (UID: "39ddf73a-4de1-404a-8a9a-276804aa9209"). InnerVolumeSpecName "kube-api-access-sn6sg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:55:02.558278 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:02.558249 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ddf73a-4de1-404a-8a9a-276804aa9209-dshm" pod "39ddf73a-4de1-404a-8a9a-276804aa9209" (UID: "39ddf73a-4de1-404a-8a9a-276804aa9209"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:55:02.657500 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:02.657443 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39ddf73a-4de1-404a-8a9a-276804aa9209-tls-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:55:02.657500 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:02.657468 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn6sg\" (UniqueName: \"kubernetes.io/projected/39ddf73a-4de1-404a-8a9a-276804aa9209-kube-api-access-sn6sg\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:55:02.657500 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:02.657479 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ddf73a-4de1-404a-8a9a-276804aa9209-dshm\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:55:02.657500 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:02.657489 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ddf73a-4de1-404a-8a9a-276804aa9209-tmp-dir\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:55:02.984500 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:02.984419 4952 generic.go:356] "Generic (PLEG): container finished" podID="39ddf73a-4de1-404a-8a9a-276804aa9209" containerID="d4eec5bb1d2355560ecfcf0977b4615951cb6fbdbf7c05b784c4597b0c69ab12" exitCode=0 Oct 30 23:55:02.984500 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:02.984488 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq" Oct 30 23:55:02.985064 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:02.984505 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq" event={"ID":"39ddf73a-4de1-404a-8a9a-276804aa9209","Type":"ContainerDied","Data":"d4eec5bb1d2355560ecfcf0977b4615951cb6fbdbf7c05b784c4597b0c69ab12"} Oct 30 23:55:02.985064 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:02.984543 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq" event={"ID":"39ddf73a-4de1-404a-8a9a-276804aa9209","Type":"ContainerDied","Data":"ac6241d5b8d106c9f1a9e3537cfea960fff3fb9da828497d85079861284e74ae"} Oct 30 23:55:02.985064 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:02.984560 4952 scope.go:122] "RemoveContainer" containerID="d4eec5bb1d2355560ecfcf0977b4615951cb6fbdbf7c05b784c4597b0c69ab12" Oct 30 23:55:02.995474 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:02.995459 4952 scope.go:122] "RemoveContainer" containerID="d4eec5bb1d2355560ecfcf0977b4615951cb6fbdbf7c05b784c4597b0c69ab12" Oct 30 23:55:02.995721 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:55:02.995703 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4eec5bb1d2355560ecfcf0977b4615951cb6fbdbf7c05b784c4597b0c69ab12\": container with ID starting with d4eec5bb1d2355560ecfcf0977b4615951cb6fbdbf7c05b784c4597b0c69ab12 not found: ID does not exist" containerID="d4eec5bb1d2355560ecfcf0977b4615951cb6fbdbf7c05b784c4597b0c69ab12" Oct 30 23:55:02.995790 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:02.995728 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4eec5bb1d2355560ecfcf0977b4615951cb6fbdbf7c05b784c4597b0c69ab12"} err="failed to get container status \"d4eec5bb1d2355560ecfcf0977b4615951cb6fbdbf7c05b784c4597b0c69ab12\": rpc error: code = NotFound desc = could not find container \"d4eec5bb1d2355560ecfcf0977b4615951cb6fbdbf7c05b784c4597b0c69ab12\": container with ID starting with d4eec5bb1d2355560ecfcf0977b4615951cb6fbdbf7c05b784c4597b0c69ab12 not found: ID does not exist" Oct 30 23:55:03.152725 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:03.152687 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq"] Oct 30 23:55:03.155233 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:03.155211 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-71a68a/test-model-71a68a-kserve-78ff84d5b7-5srmq"] Oct 30 23:55:04.970151 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:04.970117 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39ddf73a-4de1-404a-8a9a-276804aa9209" path="/var/lib/kubelet/pods/39ddf73a-4de1-404a-8a9a-276804aa9209/volumes" Oct 30 23:55:05.817619 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:05.817588 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk"] Oct 30 23:55:05.817816 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:05.817792 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk" podUID="dd4ad911-4b05-4bf3-82c4-9c368ce50137" containerName="maas-api" containerID="cri-o://4ed8bfbcf1cd8dc3f73b752465e6002016defb78b15728e9a5b666be4b2d559f" gracePeriod=30 Oct 30 23:55:06.014177 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:06.014146 4952 generic.go:356] "Generic (PLEG): container finished" podID="dd4ad911-4b05-4bf3-82c4-9c368ce50137" containerID="4ed8bfbcf1cd8dc3f73b752465e6002016defb78b15728e9a5b666be4b2d559f" exitCode=0 Oct 30 23:55:06.014518 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:06.014219 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk" event={"ID":"dd4ad911-4b05-4bf3-82c4-9c368ce50137","Type":"ContainerDied","Data":"4ed8bfbcf1cd8dc3f73b752465e6002016defb78b15728e9a5b666be4b2d559f"} Oct 30 23:55:06.046512 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:06.046477 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:06.097210 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:06.097182 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk" Oct 30 23:55:06.192721 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:06.192685 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd4ad911-4b05-4bf3-82c4-9c368ce50137-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd4ad911-4b05-4bf3-82c4-9c368ce50137-maas-api-tls\") pod \"dd4ad911-4b05-4bf3-82c4-9c368ce50137\" (UID: \"dd4ad911-4b05-4bf3-82c4-9c368ce50137\") " Oct 30 23:55:06.192938 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:06.192747 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd4ad911-4b05-4bf3-82c4-9c368ce50137-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd4ad911-4b05-4bf3-82c4-9c368ce50137-metrics-certs\") pod \"dd4ad911-4b05-4bf3-82c4-9c368ce50137\" (UID: \"dd4ad911-4b05-4bf3-82c4-9c368ce50137\") " Oct 30 23:55:06.192938 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:06.192833 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd4ad911-4b05-4bf3-82c4-9c368ce50137-kube-api-access-82q2d\" (UniqueName: \"kubernetes.io/projected/dd4ad911-4b05-4bf3-82c4-9c368ce50137-kube-api-access-82q2d\") pod \"dd4ad911-4b05-4bf3-82c4-9c368ce50137\" (UID: \"dd4ad911-4b05-4bf3-82c4-9c368ce50137\") " Oct 30 23:55:06.193115 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:06.192981 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd4ad911-4b05-4bf3-82c4-9c368ce50137-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd4ad911-4b05-4bf3-82c4-9c368ce50137-openshift-service-ca\") pod \"dd4ad911-4b05-4bf3-82c4-9c368ce50137\" (UID: \"dd4ad911-4b05-4bf3-82c4-9c368ce50137\") " Oct 30 23:55:06.193465 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:06.193402 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd4ad911-4b05-4bf3-82c4-9c368ce50137-openshift-service-ca" pod "dd4ad911-4b05-4bf3-82c4-9c368ce50137" (UID: "dd4ad911-4b05-4bf3-82c4-9c368ce50137"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:55:06.193760 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:06.193738 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd4ad911-4b05-4bf3-82c4-9c368ce50137-openshift-service-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:55:06.195288 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:06.195259 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd4ad911-4b05-4bf3-82c4-9c368ce50137-metrics-certs" pod "dd4ad911-4b05-4bf3-82c4-9c368ce50137" (UID: "dd4ad911-4b05-4bf3-82c4-9c368ce50137"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:55:06.195370 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:06.195288 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd4ad911-4b05-4bf3-82c4-9c368ce50137-kube-api-access-82q2d" pod "dd4ad911-4b05-4bf3-82c4-9c368ce50137" (UID: "dd4ad911-4b05-4bf3-82c4-9c368ce50137"). InnerVolumeSpecName "kube-api-access-82q2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:55:06.195495 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:06.195478 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd4ad911-4b05-4bf3-82c4-9c368ce50137-maas-api-tls" pod "dd4ad911-4b05-4bf3-82c4-9c368ce50137" (UID: "dd4ad911-4b05-4bf3-82c4-9c368ce50137"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:55:06.295193 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:06.295162 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd4ad911-4b05-4bf3-82c4-9c368ce50137-maas-api-tls\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:55:06.295193 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:06.295191 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd4ad911-4b05-4bf3-82c4-9c368ce50137-metrics-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:55:06.295193 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:06.295201 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82q2d\" (UniqueName: \"kubernetes.io/projected/dd4ad911-4b05-4bf3-82c4-9c368ce50137-kube-api-access-82q2d\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:55:07.021762 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:07.021734 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk" Oct 30 23:55:07.021762 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:07.021747 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk" event={"ID":"dd4ad911-4b05-4bf3-82c4-9c368ce50137","Type":"ContainerDied","Data":"2f480e2053565f8f31e141d50ea53e7804f2dd938f15260eb5ef2e5b056b8b0e"} Oct 30 23:55:07.022196 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:07.021807 4952 scope.go:122] "RemoveContainer" containerID="4ed8bfbcf1cd8dc3f73b752465e6002016defb78b15728e9a5b666be4b2d559f" Oct 30 23:55:07.198130 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:07.198099 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk"] Oct 30 23:55:07.202597 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:07.202568 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:07.202761 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:07.202663 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2cfb80-84f6d84985-tpgqk"] Oct 30 23:55:07.579791 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:07.579756 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:07.860792 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:07.860758 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:08.677145 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:08.677113 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:08.927121 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:08.927079 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:08.970398 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:08.970330 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd4ad911-4b05-4bf3-82c4-9c368ce50137" path="/var/lib/kubelet/pods/dd4ad911-4b05-4bf3-82c4-9c368ce50137/volumes" Oct 30 23:55:09.241736 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:09.241645 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:09.950827 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:09.950791 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:14.311374 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:14.311338 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:15.210840 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:15.210793 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:15.475753 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:15.475661 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:15.743989 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:15.743886 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:16.849216 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:16.849173 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:17.154333 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:17.154226 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:17.464336 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:17.464250 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:19.449215 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:19.449174 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:21.142080 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:55:21.142027 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:55:21.835863 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:21.835828 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:22.080806 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:22.080766 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:24.585449 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:24.585417 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:27.264845 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:27.264814 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:27.482624 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:27.482586 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:29.491822 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:29.491783 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:30.024624 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:30.024592 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:30.292804 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:30.292694 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:34.394587 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:34.394554 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:39.269384 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:39.269353 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:39.560911 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:39.560834 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:55:44.110289 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:44.110242 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:48.868414 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:48.868381 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:51.148746 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:55:51.144237 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:55:53.729472 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:53.729435 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:55:58.817600 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:55:58.817566 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:03.136510 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:03.136467 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:04.078296 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:04.078266 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k"] Oct 30 23:56:04.276847 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:04.276794 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:04.371555 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:04.371490 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k" podUID="d534649f-0944-4efd-b936-069af23b30ba" containerName="maas-api" containerID="cri-o://af1d771f0be7e6dcada89dac0a154b5b1217edd731ce0ef30bc3864530280c95" gracePeriod=30 Oct 30 23:56:04.638534 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:04.638477 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k" Oct 30 23:56:04.658902 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:04.658878 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:04.673316 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:04.673291 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d534649f-0944-4efd-b936-069af23b30ba-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d534649f-0944-4efd-b936-069af23b30ba-openshift-service-ca\") pod \"d534649f-0944-4efd-b936-069af23b30ba\" (UID: \"d534649f-0944-4efd-b936-069af23b30ba\") " Oct 30 23:56:04.673447 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:04.673433 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d534649f-0944-4efd-b936-069af23b30ba-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d534649f-0944-4efd-b936-069af23b30ba-maas-api-tls\") pod \"d534649f-0944-4efd-b936-069af23b30ba\" (UID: \"d534649f-0944-4efd-b936-069af23b30ba\") " Oct 30 23:56:04.673517 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:04.673460 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d534649f-0944-4efd-b936-069af23b30ba-kube-api-access-klw7c\" (UniqueName: \"kubernetes.io/projected/d534649f-0944-4efd-b936-069af23b30ba-kube-api-access-klw7c\") pod \"d534649f-0944-4efd-b936-069af23b30ba\" (UID: \"d534649f-0944-4efd-b936-069af23b30ba\") " Oct 30 23:56:04.673517 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:04.673479 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d534649f-0944-4efd-b936-069af23b30ba-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d534649f-0944-4efd-b936-069af23b30ba-metrics-certs\") pod \"d534649f-0944-4efd-b936-069af23b30ba\" (UID: \"d534649f-0944-4efd-b936-069af23b30ba\") " Oct 30 23:56:04.673695 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:04.673674 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d534649f-0944-4efd-b936-069af23b30ba-openshift-service-ca" pod "d534649f-0944-4efd-b936-069af23b30ba" (UID: "d534649f-0944-4efd-b936-069af23b30ba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:56:04.673801 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:04.673784 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d534649f-0944-4efd-b936-069af23b30ba-openshift-service-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:56:04.675711 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:04.675689 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d534649f-0944-4efd-b936-069af23b30ba-maas-api-tls" pod "d534649f-0944-4efd-b936-069af23b30ba" (UID: "d534649f-0944-4efd-b936-069af23b30ba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:56:04.675791 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:04.675709 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d534649f-0944-4efd-b936-069af23b30ba-kube-api-access-klw7c" pod "d534649f-0944-4efd-b936-069af23b30ba" (UID: "d534649f-0944-4efd-b936-069af23b30ba"). InnerVolumeSpecName "kube-api-access-klw7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:56:04.675791 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:04.675727 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d534649f-0944-4efd-b936-069af23b30ba-metrics-certs" pod "d534649f-0944-4efd-b936-069af23b30ba" (UID: "d534649f-0944-4efd-b936-069af23b30ba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:56:04.774650 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:04.774619 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d534649f-0944-4efd-b936-069af23b30ba-maas-api-tls\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:56:04.774800 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:04.774656 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klw7c\" (UniqueName: \"kubernetes.io/projected/d534649f-0944-4efd-b936-069af23b30ba-kube-api-access-klw7c\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:56:04.774800 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:04.774672 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d534649f-0944-4efd-b936-069af23b30ba-metrics-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:56:04.947149 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:04.947064 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:05.379540 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:05.379502 4952 generic.go:356] "Generic (PLEG): container finished" podID="d534649f-0944-4efd-b936-069af23b30ba" containerID="af1d771f0be7e6dcada89dac0a154b5b1217edd731ce0ef30bc3864530280c95" exitCode=0 Oct 30 23:56:05.380026 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:05.379551 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k" Oct 30 23:56:05.380026 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:05.379578 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k" event={"ID":"d534649f-0944-4efd-b936-069af23b30ba","Type":"ContainerDied","Data":"af1d771f0be7e6dcada89dac0a154b5b1217edd731ce0ef30bc3864530280c95"} Oct 30 23:56:05.380026 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:05.379652 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k" event={"ID":"d534649f-0944-4efd-b936-069af23b30ba","Type":"ContainerDied","Data":"fae9a2a4b25c4ff64351ed6433d2c62cc856f9a9595c8413f7e2948f06d65ec2"} Oct 30 23:56:05.380026 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:05.379683 4952 scope.go:122] "RemoveContainer" containerID="af1d771f0be7e6dcada89dac0a154b5b1217edd731ce0ef30bc3864530280c95" Oct 30 23:56:05.389889 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:05.389870 4952 scope.go:122] "RemoveContainer" containerID="af1d771f0be7e6dcada89dac0a154b5b1217edd731ce0ef30bc3864530280c95" Oct 30 23:56:05.390236 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:56:05.390211 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af1d771f0be7e6dcada89dac0a154b5b1217edd731ce0ef30bc3864530280c95\": container with ID starting with af1d771f0be7e6dcada89dac0a154b5b1217edd731ce0ef30bc3864530280c95 not found: ID does not exist" containerID="af1d771f0be7e6dcada89dac0a154b5b1217edd731ce0ef30bc3864530280c95" Oct 30 23:56:05.390343 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:05.390245 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af1d771f0be7e6dcada89dac0a154b5b1217edd731ce0ef30bc3864530280c95"} err="failed to get container status \"af1d771f0be7e6dcada89dac0a154b5b1217edd731ce0ef30bc3864530280c95\": rpc error: code = NotFound desc = could not find container \"af1d771f0be7e6dcada89dac0a154b5b1217edd731ce0ef30bc3864530280c95\": container with ID starting with af1d771f0be7e6dcada89dac0a154b5b1217edd731ce0ef30bc3864530280c95 not found: ID does not exist" Oct 30 23:56:05.544061 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:05.544013 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k"] Oct 30 23:56:05.593398 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:05.593369 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-15b137-7cf4fd9c7d-z6d5k"] Oct 30 23:56:06.625357 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:06.625322 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:06.868517 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:06.868488 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:06.970265 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:06.970195 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d534649f-0944-4efd-b936-069af23b30ba" path="/var/lib/kubelet/pods/d534649f-0944-4efd-b936-069af23b30ba/volumes" Oct 30 23:56:10.730520 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:10.730489 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:11.583771 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:11.583738 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:11.823324 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:11.823296 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:12.047165 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:12.047134 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm"] Oct 30 23:56:12.047357 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:12.047333 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" podUID="c1d92b1e-2565-4881-ae9e-c605e91aa1b8" containerName="istio-proxy" containerID="cri-o://b50f76e5c0db692dc142780d07e8ccc16be9e0d0dc553f697d3f42981e1d8e54" gracePeriod=30 Oct 30 23:56:12.066986 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:12.066911 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:12.340713 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:12.340676 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:13.417242 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:13.417207 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:56:13.625871 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:13.625832 4952 patch_prober.go:29] interesting pod/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 23:56:13.626114 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:13.625910 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" podUID="c1d92b1e-2565-4881-ae9e-c605e91aa1b8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:56:14.489502 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:14.489468 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:14.747898 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:14.747821 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:15.039140 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:15.039024 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:17.154292 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.154223 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:17.323011 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.322933 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" Oct 30 23:56:17.395174 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.395146 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-istio-data\") pod \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " Oct 30 23:56:17.395332 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.395259 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-istio-envoy\") pod \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " Oct 30 23:56:17.395332 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.395283 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-kube-api-access-8t6s9\" (UniqueName: \"kubernetes.io/projected/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-kube-api-access-8t6s9\") pod \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " Oct 30 23:56:17.395332 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.395317 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-workload-socket\") pod \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " Oct 30 23:56:17.395469 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.395383 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-istio-data" pod "c1d92b1e-2565-4881-ae9e-c605e91aa1b8" (UID: "c1d92b1e-2565-4881-ae9e-c605e91aa1b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:56:17.395469 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.395425 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-istio-podinfo\") pod \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " Oct 30 23:56:17.395469 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.395456 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-workload-certs\") pod \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " Oct 30 23:56:17.395583 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.395516 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-credential-socket\") pod \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " Oct 30 23:56:17.395629 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.395579 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-istiod-ca-cert\") pod \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " Oct 30 23:56:17.395629 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.395592 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-workload-socket" pod "c1d92b1e-2565-4881-ae9e-c605e91aa1b8" (UID: "c1d92b1e-2565-4881-ae9e-c605e91aa1b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:56:17.395629 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.395609 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-istio-token\" (UniqueName: \"kubernetes.io/projected/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-istio-token\") pod \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\" (UID: \"c1d92b1e-2565-4881-ae9e-c605e91aa1b8\") " Oct 30 23:56:17.395793 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.395742 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-workload-certs" pod "c1d92b1e-2565-4881-ae9e-c605e91aa1b8" (UID: "c1d92b1e-2565-4881-ae9e-c605e91aa1b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:56:17.396074 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.395802 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-credential-socket" pod "c1d92b1e-2565-4881-ae9e-c605e91aa1b8" (UID: "c1d92b1e-2565-4881-ae9e-c605e91aa1b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:56:17.396074 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.395978 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-istiod-ca-cert" pod "c1d92b1e-2565-4881-ae9e-c605e91aa1b8" (UID: "c1d92b1e-2565-4881-ae9e-c605e91aa1b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:56:17.396242 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.396135 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-workload-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:56:17.396242 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.396156 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-workload-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:56:17.396242 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.396170 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-istio-data\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:56:17.397693 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.397666 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-kube-api-access-8t6s9" pod "c1d92b1e-2565-4881-ae9e-c605e91aa1b8" (UID: "c1d92b1e-2565-4881-ae9e-c605e91aa1b8"). InnerVolumeSpecName "kube-api-access-8t6s9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:56:17.398152 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.398125 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-istio-token" pod "c1d92b1e-2565-4881-ae9e-c605e91aa1b8" (UID: "c1d92b1e-2565-4881-ae9e-c605e91aa1b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:56:17.398255 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.398196 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-istio-podinfo" pod "c1d92b1e-2565-4881-ae9e-c605e91aa1b8" (UID: "c1d92b1e-2565-4881-ae9e-c605e91aa1b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:56:17.398255 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.398196 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-istio-envoy" pod "c1d92b1e-2565-4881-ae9e-c605e91aa1b8" (UID: "c1d92b1e-2565-4881-ae9e-c605e91aa1b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:56:17.453882 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.453857 4952 generic.go:356] "Generic (PLEG): container finished" podID="c1d92b1e-2565-4881-ae9e-c605e91aa1b8" containerID="b50f76e5c0db692dc142780d07e8ccc16be9e0d0dc553f697d3f42981e1d8e54" exitCode=0 Oct 30 23:56:17.453999 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.453905 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" Oct 30 23:56:17.453999 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.453945 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" event={"ID":"c1d92b1e-2565-4881-ae9e-c605e91aa1b8","Type":"ContainerDied","Data":"b50f76e5c0db692dc142780d07e8ccc16be9e0d0dc553f697d3f42981e1d8e54"} Oct 30 23:56:17.454129 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.454008 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm" event={"ID":"c1d92b1e-2565-4881-ae9e-c605e91aa1b8","Type":"ContainerDied","Data":"ec18df05ca663040f4fe74b9c8dcf4aeb1c5fb18a7eb3fc2611cfc191e0e1b61"} Oct 30 23:56:17.454129 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.454028 4952 scope.go:122] "RemoveContainer" containerID="b50f76e5c0db692dc142780d07e8ccc16be9e0d0dc553f697d3f42981e1d8e54" Oct 30 23:56:17.464675 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.464660 4952 scope.go:122] "RemoveContainer" containerID="b50f76e5c0db692dc142780d07e8ccc16be9e0d0dc553f697d3f42981e1d8e54" Oct 30 23:56:17.464914 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:56:17.464894 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b50f76e5c0db692dc142780d07e8ccc16be9e0d0dc553f697d3f42981e1d8e54\": container with ID starting with b50f76e5c0db692dc142780d07e8ccc16be9e0d0dc553f697d3f42981e1d8e54 not found: ID does not exist" containerID="b50f76e5c0db692dc142780d07e8ccc16be9e0d0dc553f697d3f42981e1d8e54" Oct 30 23:56:17.464974 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.464920 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b50f76e5c0db692dc142780d07e8ccc16be9e0d0dc553f697d3f42981e1d8e54"} err="failed to get container status \"b50f76e5c0db692dc142780d07e8ccc16be9e0d0dc553f697d3f42981e1d8e54\": rpc error: code = NotFound desc = could not find container \"b50f76e5c0db692dc142780d07e8ccc16be9e0d0dc553f697d3f42981e1d8e54\": container with ID starting with b50f76e5c0db692dc142780d07e8ccc16be9e0d0dc553f697d3f42981e1d8e54 not found: ID does not exist" Oct 30 23:56:17.497242 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.496829 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-istio-envoy\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:56:17.497242 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.496860 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t6s9\" (UniqueName: \"kubernetes.io/projected/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-kube-api-access-8t6s9\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:56:17.497242 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.496876 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-istio-podinfo\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:56:17.497242 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.496884 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-credential-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:56:17.497242 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.496894 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-istiod-ca-cert\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:56:17.497242 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.496906 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d92b1e-2565-4881-ae9e-c605e91aa1b8-istio-token\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:56:17.660008 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.659978 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm"] Oct 30 23:56:17.665558 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:17.665534 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-zfhlm"] Oct 30 23:56:18.546655 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:18.546620 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:18.969767 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:18.969731 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1d92b1e-2565-4881-ae9e-c605e91aa1b8" path="/var/lib/kubelet/pods/c1d92b1e-2565-4881-ae9e-c605e91aa1b8/volumes" Oct 30 23:56:20.926498 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:20.926459 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:21.141149 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:56:21.141111 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:56:21.447547 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:21.447518 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft_91651ce2-4673-4ced-b26d-e94ff8829d11/extract/0.log" Oct 30 23:56:21.457586 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:21.457565 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/2.log" Oct 30 23:56:21.462452 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:21.462433 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft_91651ce2-4673-4ced-b26d-e94ff8829d11/extract/0.log" Oct 30 23:56:21.463573 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:21.463553 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 30 23:56:21.472661 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:21.472639 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/2.log" Oct 30 23:56:21.486423 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:21.486407 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 30 23:56:22.111536 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:22.111505 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:24.517182 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:24.517147 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:24.757951 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:24.757922 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:26.265538 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:26.265479 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:26.781390 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:26.781353 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:27.016168 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:27.016137 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:27.627268 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:27.627232 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:28.132587 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:28.132557 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:30.849419 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:30.849308 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:31.139665 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:31.139587 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:31.791294 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:31.791260 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:33.184563 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:33.184535 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:35.630604 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:35.630567 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:37.454680 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:37.454646 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:56:37.909919 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:37.909877 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8"] Oct 30 23:56:37.910178 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:37.910129 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" podUID="323943ac-0c8a-4612-84d0-3e009c8436ab" containerName="istio-proxy" containerID="cri-o://88238ded6d97e7f08a0a80cf8395c3f048946128803a307141b82a7844d8398b" gracePeriod=30 Oct 30 23:56:38.237191 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:38.237077 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-rnmq8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 23:56:38.237339 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:38.237184 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" podUID="323943ac-0c8a-4612-84d0-3e009c8436ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:56:38.631175 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:38.631143 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:38.896063 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:38.895954 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:39.207593 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:39.207508 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:43.175583 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.175555 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" Oct 30 23:56:43.253891 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.253833 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/323943ac-0c8a-4612-84d0-3e009c8436ab-kube-api-access-h2zv9\" (UniqueName: \"kubernetes.io/projected/323943ac-0c8a-4612-84d0-3e009c8436ab-kube-api-access-h2zv9\") pod \"323943ac-0c8a-4612-84d0-3e009c8436ab\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " Oct 30 23:56:43.253891 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.253863 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323943ac-0c8a-4612-84d0-3e009c8436ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323943ac-0c8a-4612-84d0-3e009c8436ab-workload-socket\") pod \"323943ac-0c8a-4612-84d0-3e009c8436ab\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " Oct 30 23:56:43.254184 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.253898 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/323943ac-0c8a-4612-84d0-3e009c8436ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323943ac-0c8a-4612-84d0-3e009c8436ab-istio-podinfo\") pod \"323943ac-0c8a-4612-84d0-3e009c8436ab\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " Oct 30 23:56:43.254184 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.253942 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323943ac-0c8a-4612-84d0-3e009c8436ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323943ac-0c8a-4612-84d0-3e009c8436ab-credential-socket\") pod \"323943ac-0c8a-4612-84d0-3e009c8436ab\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " Oct 30 23:56:43.254184 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.253968 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/323943ac-0c8a-4612-84d0-3e009c8436ab-istio-token\" (UniqueName: \"kubernetes.io/projected/323943ac-0c8a-4612-84d0-3e009c8436ab-istio-token\") pod \"323943ac-0c8a-4612-84d0-3e009c8436ab\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " Oct 30 23:56:43.254184 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.254001 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323943ac-0c8a-4612-84d0-3e009c8436ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323943ac-0c8a-4612-84d0-3e009c8436ab-istio-data\") pod \"323943ac-0c8a-4612-84d0-3e009c8436ab\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " Oct 30 23:56:43.254184 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.254108 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/323943ac-0c8a-4612-84d0-3e009c8436ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323943ac-0c8a-4612-84d0-3e009c8436ab-istiod-ca-cert\") pod \"323943ac-0c8a-4612-84d0-3e009c8436ab\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " Oct 30 23:56:43.254184 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.254139 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323943ac-0c8a-4612-84d0-3e009c8436ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323943ac-0c8a-4612-84d0-3e009c8436ab-workload-certs\") pod \"323943ac-0c8a-4612-84d0-3e009c8436ab\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " Oct 30 23:56:43.254528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.254195 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323943ac-0c8a-4612-84d0-3e009c8436ab-workload-socket" pod "323943ac-0c8a-4612-84d0-3e009c8436ab" (UID: "323943ac-0c8a-4612-84d0-3e009c8436ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:56:43.254528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.254215 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323943ac-0c8a-4612-84d0-3e009c8436ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323943ac-0c8a-4612-84d0-3e009c8436ab-istio-envoy\") pod \"323943ac-0c8a-4612-84d0-3e009c8436ab\" (UID: \"323943ac-0c8a-4612-84d0-3e009c8436ab\") " Oct 30 23:56:43.254528 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.254514 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323943ac-0c8a-4612-84d0-3e009c8436ab-credential-socket" pod "323943ac-0c8a-4612-84d0-3e009c8436ab" (UID: "323943ac-0c8a-4612-84d0-3e009c8436ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:56:43.254788 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.254683 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323943ac-0c8a-4612-84d0-3e009c8436ab-istio-data" pod "323943ac-0c8a-4612-84d0-3e009c8436ab" (UID: "323943ac-0c8a-4612-84d0-3e009c8436ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:56:43.254941 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.254835 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323943ac-0c8a-4612-84d0-3e009c8436ab-workload-certs" pod "323943ac-0c8a-4612-84d0-3e009c8436ab" (UID: "323943ac-0c8a-4612-84d0-3e009c8436ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:56:43.254941 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.254920 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/323943ac-0c8a-4612-84d0-3e009c8436ab-istiod-ca-cert" pod "323943ac-0c8a-4612-84d0-3e009c8436ab" (UID: "323943ac-0c8a-4612-84d0-3e009c8436ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:56:43.255163 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.255141 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323943ac-0c8a-4612-84d0-3e009c8436ab-workload-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:56:43.255256 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.255171 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323943ac-0c8a-4612-84d0-3e009c8436ab-credential-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:56:43.255256 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.255186 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323943ac-0c8a-4612-84d0-3e009c8436ab-istio-data\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:56:43.255256 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.255199 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323943ac-0c8a-4612-84d0-3e009c8436ab-istiod-ca-cert\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:56:43.255256 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.255213 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323943ac-0c8a-4612-84d0-3e009c8436ab-workload-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:56:43.256481 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.256457 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/323943ac-0c8a-4612-84d0-3e009c8436ab-kube-api-access-h2zv9" pod "323943ac-0c8a-4612-84d0-3e009c8436ab" (UID: "323943ac-0c8a-4612-84d0-3e009c8436ab"). InnerVolumeSpecName "kube-api-access-h2zv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:56:43.256855 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.256829 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/323943ac-0c8a-4612-84d0-3e009c8436ab-istio-token" pod "323943ac-0c8a-4612-84d0-3e009c8436ab" (UID: "323943ac-0c8a-4612-84d0-3e009c8436ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:56:43.257007 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.256993 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/323943ac-0c8a-4612-84d0-3e009c8436ab-istio-podinfo" pod "323943ac-0c8a-4612-84d0-3e009c8436ab" (UID: "323943ac-0c8a-4612-84d0-3e009c8436ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:56:43.257204 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.257186 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323943ac-0c8a-4612-84d0-3e009c8436ab-istio-envoy" pod "323943ac-0c8a-4612-84d0-3e009c8436ab" (UID: "323943ac-0c8a-4612-84d0-3e009c8436ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:56:43.356006 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.355979 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323943ac-0c8a-4612-84d0-3e009c8436ab-istio-podinfo\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:56:43.356006 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.356004 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323943ac-0c8a-4612-84d0-3e009c8436ab-istio-token\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:56:43.356254 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.356013 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323943ac-0c8a-4612-84d0-3e009c8436ab-istio-envoy\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:56:43.356254 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.356024 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2zv9\" (UniqueName: \"kubernetes.io/projected/323943ac-0c8a-4612-84d0-3e009c8436ab-kube-api-access-h2zv9\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:56:43.622939 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.622906 4952 generic.go:356] "Generic (PLEG): container finished" podID="323943ac-0c8a-4612-84d0-3e009c8436ab" containerID="88238ded6d97e7f08a0a80cf8395c3f048946128803a307141b82a7844d8398b" exitCode=0 Oct 30 23:56:43.623164 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.622963 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" Oct 30 23:56:43.623164 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.622965 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" event={"ID":"323943ac-0c8a-4612-84d0-3e009c8436ab","Type":"ContainerDied","Data":"88238ded6d97e7f08a0a80cf8395c3f048946128803a307141b82a7844d8398b"} Oct 30 23:56:43.623164 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.623108 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8" event={"ID":"323943ac-0c8a-4612-84d0-3e009c8436ab","Type":"ContainerDied","Data":"d4cf034f8a1036fd52f8133b940b9bc8f7ad060775f365fb05a27621ff17f28f"} Oct 30 23:56:43.623164 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.623143 4952 scope.go:122] "RemoveContainer" containerID="88238ded6d97e7f08a0a80cf8395c3f048946128803a307141b82a7844d8398b" Oct 30 23:56:43.635214 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.635197 4952 scope.go:122] "RemoveContainer" containerID="88238ded6d97e7f08a0a80cf8395c3f048946128803a307141b82a7844d8398b" Oct 30 23:56:43.635505 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:56:43.635481 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88238ded6d97e7f08a0a80cf8395c3f048946128803a307141b82a7844d8398b\": container with ID starting with 88238ded6d97e7f08a0a80cf8395c3f048946128803a307141b82a7844d8398b not found: ID does not exist" containerID="88238ded6d97e7f08a0a80cf8395c3f048946128803a307141b82a7844d8398b" Oct 30 23:56:43.635587 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.635512 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88238ded6d97e7f08a0a80cf8395c3f048946128803a307141b82a7844d8398b"} err="failed to get container status \"88238ded6d97e7f08a0a80cf8395c3f048946128803a307141b82a7844d8398b\": rpc error: code = NotFound desc = could not find container \"88238ded6d97e7f08a0a80cf8395c3f048946128803a307141b82a7844d8398b\": container with ID starting with 88238ded6d97e7f08a0a80cf8395c3f048946128803a307141b82a7844d8398b not found: ID does not exist" Oct 30 23:56:43.779250 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.779197 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:43.791103 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.791072 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8"] Oct 30 23:56:43.840481 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:43.840454 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rnmq8"] Oct 30 23:56:44.970056 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:44.970004 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="323943ac-0c8a-4612-84d0-3e009c8436ab" path="/var/lib/kubelet/pods/323943ac-0c8a-4612-84d0-3e009c8436ab/volumes" Oct 30 23:56:48.264309 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:48.264275 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:50.529002 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:50.528970 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:50.871427 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:50.871398 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:51.127134 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:51.127058 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:51.141551 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:56:51.141516 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:56:53.726930 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:53.726882 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:58.272262 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:58.272218 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:59.239668 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:59.239631 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w"] Oct 30 23:56:59.239898 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:59.239871 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w" podUID="ccd7384b-d043-4f0f-b4e8-efdcea747317" containerName="maas-api" containerID="cri-o://034028d63180aad49b8f11794139321d61a33044fac9487e1cfdc10e616c227d" gracePeriod=30 Oct 30 23:56:59.244728 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:59.244700 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w" podUID="ccd7384b-d043-4f0f-b4e8-efdcea747317" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.130:8443/health\": dial tcp 10.132.0.130:8443: connect: connection refused" Oct 30 23:56:59.509434 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:59.509405 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w" Oct 30 23:56:59.534106 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:59.534028 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:56:59.602790 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:59.602767 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccd7384b-d043-4f0f-b4e8-efdcea747317-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccd7384b-d043-4f0f-b4e8-efdcea747317-maas-api-tls\") pod \"ccd7384b-d043-4f0f-b4e8-efdcea747317\" (UID: \"ccd7384b-d043-4f0f-b4e8-efdcea747317\") " Oct 30 23:56:59.602924 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:59.602813 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccd7384b-d043-4f0f-b4e8-efdcea747317-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccd7384b-d043-4f0f-b4e8-efdcea747317-metrics-certs\") pod \"ccd7384b-d043-4f0f-b4e8-efdcea747317\" (UID: \"ccd7384b-d043-4f0f-b4e8-efdcea747317\") " Oct 30 23:56:59.602924 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:59.602860 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd7384b-d043-4f0f-b4e8-efdcea747317-kube-api-access-nl2fv\" (UniqueName: \"kubernetes.io/projected/ccd7384b-d043-4f0f-b4e8-efdcea747317-kube-api-access-nl2fv\") pod \"ccd7384b-d043-4f0f-b4e8-efdcea747317\" (UID: \"ccd7384b-d043-4f0f-b4e8-efdcea747317\") " Oct 30 23:56:59.603005 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:59.602961 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccd7384b-d043-4f0f-b4e8-efdcea747317-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccd7384b-d043-4f0f-b4e8-efdcea747317-openshift-service-ca\") pod \"ccd7384b-d043-4f0f-b4e8-efdcea747317\" (UID: \"ccd7384b-d043-4f0f-b4e8-efdcea747317\") " Oct 30 23:56:59.603313 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:59.603286 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccd7384b-d043-4f0f-b4e8-efdcea747317-openshift-service-ca" pod "ccd7384b-d043-4f0f-b4e8-efdcea747317" (UID: "ccd7384b-d043-4f0f-b4e8-efdcea747317"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:56:59.605201 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:59.605094 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccd7384b-d043-4f0f-b4e8-efdcea747317-metrics-certs" pod "ccd7384b-d043-4f0f-b4e8-efdcea747317" (UID: "ccd7384b-d043-4f0f-b4e8-efdcea747317"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:56:59.605281 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:59.605231 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccd7384b-d043-4f0f-b4e8-efdcea747317-maas-api-tls" pod "ccd7384b-d043-4f0f-b4e8-efdcea747317" (UID: "ccd7384b-d043-4f0f-b4e8-efdcea747317"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:56:59.605281 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:59.605260 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd7384b-d043-4f0f-b4e8-efdcea747317-kube-api-access-nl2fv" pod "ccd7384b-d043-4f0f-b4e8-efdcea747317" (UID: "ccd7384b-d043-4f0f-b4e8-efdcea747317"). InnerVolumeSpecName "kube-api-access-nl2fv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:56:59.704095 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:59.704060 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl2fv\" (UniqueName: \"kubernetes.io/projected/ccd7384b-d043-4f0f-b4e8-efdcea747317-kube-api-access-nl2fv\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:56:59.704095 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:59.704088 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccd7384b-d043-4f0f-b4e8-efdcea747317-openshift-service-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:56:59.704095 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:59.704102 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccd7384b-d043-4f0f-b4e8-efdcea747317-maas-api-tls\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:56:59.704327 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:59.704114 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccd7384b-d043-4f0f-b4e8-efdcea747317-metrics-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:56:59.727745 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:59.727715 4952 generic.go:356] "Generic (PLEG): container finished" podID="ccd7384b-d043-4f0f-b4e8-efdcea747317" containerID="034028d63180aad49b8f11794139321d61a33044fac9487e1cfdc10e616c227d" exitCode=0 Oct 30 23:56:59.727909 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:59.727767 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w" Oct 30 23:56:59.727909 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:59.727801 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w" event={"ID":"ccd7384b-d043-4f0f-b4e8-efdcea747317","Type":"ContainerDied","Data":"034028d63180aad49b8f11794139321d61a33044fac9487e1cfdc10e616c227d"} Oct 30 23:56:59.727909 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:59.727855 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w" event={"ID":"ccd7384b-d043-4f0f-b4e8-efdcea747317","Type":"ContainerDied","Data":"80a5ec38e4fa5861aad1e35b16d28ee3a4adeedb772d205163b8ce36e0e3eb5d"} Oct 30 23:56:59.727909 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:59.727882 4952 scope.go:122] "RemoveContainer" containerID="034028d63180aad49b8f11794139321d61a33044fac9487e1cfdc10e616c227d" Oct 30 23:56:59.739011 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:59.738997 4952 scope.go:122] "RemoveContainer" containerID="034028d63180aad49b8f11794139321d61a33044fac9487e1cfdc10e616c227d" Oct 30 23:56:59.739340 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:56:59.739322 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"034028d63180aad49b8f11794139321d61a33044fac9487e1cfdc10e616c227d\": container with ID starting with 034028d63180aad49b8f11794139321d61a33044fac9487e1cfdc10e616c227d not found: ID does not exist" containerID="034028d63180aad49b8f11794139321d61a33044fac9487e1cfdc10e616c227d" Oct 30 23:56:59.739401 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:59.739349 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"034028d63180aad49b8f11794139321d61a33044fac9487e1cfdc10e616c227d"} err="failed to get container status \"034028d63180aad49b8f11794139321d61a33044fac9487e1cfdc10e616c227d\": rpc error: code = NotFound desc = could not find container \"034028d63180aad49b8f11794139321d61a33044fac9487e1cfdc10e616c227d\": container with ID starting with 034028d63180aad49b8f11794139321d61a33044fac9487e1cfdc10e616c227d not found: ID does not exist" Oct 30 23:56:59.918708 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:59.918680 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w"] Oct 30 23:56:59.921739 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:59.921716 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a8204d-6749df5b5b-gjd7w"] Oct 30 23:56:59.924292 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:56:59.924273 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:00.522093 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:00.522061 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:00.888258 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:00.888227 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:00.977551 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:00.977522 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccd7384b-d043-4f0f-b4e8-efdcea747317" path="/var/lib/kubelet/pods/ccd7384b-d043-4f0f-b4e8-efdcea747317/volumes" Oct 30 23:57:01.204824 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:01.204748 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:01.491530 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:01.491448 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:02.166087 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:02.166031 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:04.969992 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:04.969963 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:57:06.091121 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:06.091087 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:07.770480 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:07.770448 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:08.037107 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:08.037005 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:08.366094 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:08.366023 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:08.709634 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:08.709557 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:08.985610 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:08.985533 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:10.179436 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:10.179399 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:10.442276 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:10.442200 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:10.696615 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:10.696541 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:11.049937 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.049856 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:11.434150 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.434107 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5dc3c0-9sdxx"] Oct 30 23:57:11.434620 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.434572 4952 cpu_manager.go:414] "containerMap: removing container" podUID="553c8b63-e975-498f-9313-878fbf99066d" containerName="istio-proxy" Oct 30 23:57:11.434620 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.434584 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="553c8b63-e975-498f-9313-878fbf99066d" containerName="istio-proxy" Oct 30 23:57:11.434620 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.434600 4952 cpu_manager.go:414] "containerMap: removing container" podUID="c1d92b1e-2565-4881-ae9e-c605e91aa1b8" containerName="istio-proxy" Oct 30 23:57:11.434620 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.434606 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1d92b1e-2565-4881-ae9e-c605e91aa1b8" containerName="istio-proxy" Oct 30 23:57:11.434835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.434627 4952 cpu_manager.go:414] "containerMap: removing container" podUID="323943ac-0c8a-4612-84d0-3e009c8436ab" containerName="istio-proxy" Oct 30 23:57:11.434835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.434633 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="323943ac-0c8a-4612-84d0-3e009c8436ab" containerName="istio-proxy" Oct 30 23:57:11.434835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.434640 4952 cpu_manager.go:414] "containerMap: removing container" podUID="da4935dd-c148-4cc4-84ab-1bfd91cce9b4" containerName="main" Oct 30 23:57:11.434835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.434644 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da4935dd-c148-4cc4-84ab-1bfd91cce9b4" containerName="main" Oct 30 23:57:11.434835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.434651 4952 cpu_manager.go:414] "containerMap: removing container" podUID="dd4ad911-4b05-4bf3-82c4-9c368ce50137" containerName="maas-api" Oct 30 23:57:11.434835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.434655 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd4ad911-4b05-4bf3-82c4-9c368ce50137" containerName="maas-api" Oct 30 23:57:11.434835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.434664 4952 cpu_manager.go:414] "containerMap: removing container" podUID="d75821f5-34cd-4eeb-a9cc-a0c19573f0f8" containerName="istio-proxy" Oct 30 23:57:11.434835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.434668 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d75821f5-34cd-4eeb-a9cc-a0c19573f0f8" containerName="istio-proxy" Oct 30 23:57:11.434835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.434675 4952 cpu_manager.go:414] "containerMap: removing container" podUID="414e5e6c-0f02-40f5-806f-0ad048215d6a" containerName="istio-proxy" Oct 30 23:57:11.434835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.434679 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="414e5e6c-0f02-40f5-806f-0ad048215d6a" containerName="istio-proxy" Oct 30 23:57:11.434835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.434685 4952 cpu_manager.go:414] "containerMap: removing container" podUID="ccd7384b-d043-4f0f-b4e8-efdcea747317" containerName="maas-api" Oct 30 23:57:11.434835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.434690 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccd7384b-d043-4f0f-b4e8-efdcea747317" containerName="maas-api" Oct 30 23:57:11.434835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.434702 4952 cpu_manager.go:414] "containerMap: removing container" podUID="305e64da-56e7-4fbb-b9fa-a7f486ed1dac" containerName="discovery" Oct 30 23:57:11.434835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.434707 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="305e64da-56e7-4fbb-b9fa-a7f486ed1dac" containerName="discovery" Oct 30 23:57:11.434835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.434714 4952 cpu_manager.go:414] "containerMap: removing container" podUID="d534649f-0944-4efd-b936-069af23b30ba" containerName="maas-api" Oct 30 23:57:11.434835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.434718 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d534649f-0944-4efd-b936-069af23b30ba" containerName="maas-api" Oct 30 23:57:11.434835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.434727 4952 cpu_manager.go:414] "containerMap: removing container" podUID="39ddf73a-4de1-404a-8a9a-276804aa9209" containerName="main" Oct 30 23:57:11.434835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.434731 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39ddf73a-4de1-404a-8a9a-276804aa9209" containerName="main" Oct 30 23:57:11.434835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.434801 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="414e5e6c-0f02-40f5-806f-0ad048215d6a" containerName="istio-proxy" Oct 30 23:57:11.434835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.434808 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccd7384b-d043-4f0f-b4e8-efdcea747317" containerName="maas-api" Oct 30 23:57:11.434835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.434817 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="305e64da-56e7-4fbb-b9fa-a7f486ed1dac" containerName="discovery" Oct 30 23:57:11.434835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.434822 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="d534649f-0944-4efd-b936-069af23b30ba" containerName="maas-api" Oct 30 23:57:11.434835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.434828 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="39ddf73a-4de1-404a-8a9a-276804aa9209" containerName="main" Oct 30 23:57:11.434835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.434836 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="553c8b63-e975-498f-9313-878fbf99066d" containerName="istio-proxy" Oct 30 23:57:11.434835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.434843 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1d92b1e-2565-4881-ae9e-c605e91aa1b8" containerName="istio-proxy" Oct 30 23:57:11.434835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.434849 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="323943ac-0c8a-4612-84d0-3e009c8436ab" containerName="istio-proxy" Oct 30 23:57:11.434835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.434854 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd4ad911-4b05-4bf3-82c4-9c368ce50137" containerName="maas-api" Oct 30 23:57:11.436171 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.434859 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="da4935dd-c148-4cc4-84ab-1bfd91cce9b4" containerName="main" Oct 30 23:57:11.436171 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.434865 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="d75821f5-34cd-4eeb-a9cc-a0c19573f0f8" containerName="istio-proxy" Oct 30 23:57:11.439307 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.439281 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5dc3c0-9sdxx" Oct 30 23:57:11.496411 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.496383 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5dc3c0-9sdxx"] Oct 30 23:57:11.542707 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.542681 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:57:11.542856 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.542738 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\"" Oct 30 23:57:11.619194 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.619163 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94dbb4c8-b2fd-4356-b458-34f860de2057-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-5dc3c0-9sdxx\" (UID: \"94dbb4c8-b2fd-4356-b458-34f860de2057\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5dc3c0-9sdxx" Oct 30 23:57:11.619361 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.619218 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94dbb4c8-b2fd-4356-b458-34f860de2057-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-5dc3c0-9sdxx\" (UID: \"94dbb4c8-b2fd-4356-b458-34f860de2057\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5dc3c0-9sdxx" Oct 30 23:57:11.619432 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.619370 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9ldc\" (UniqueName: \"kubernetes.io/projected/94dbb4c8-b2fd-4356-b458-34f860de2057-kube-api-access-x9ldc\") pod \"maas-api-revoke-keys-e2e-worker-w5-5dc3c0-9sdxx\" (UID: \"94dbb4c8-b2fd-4356-b458-34f860de2057\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5dc3c0-9sdxx" Oct 30 23:57:11.720778 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.720706 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9ldc\" (UniqueName: \"kubernetes.io/projected/94dbb4c8-b2fd-4356-b458-34f860de2057-kube-api-access-x9ldc\") pod \"maas-api-revoke-keys-e2e-worker-w5-5dc3c0-9sdxx\" (UID: \"94dbb4c8-b2fd-4356-b458-34f860de2057\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5dc3c0-9sdxx" Oct 30 23:57:11.720778 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.720752 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94dbb4c8-b2fd-4356-b458-34f860de2057-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-5dc3c0-9sdxx\" (UID: \"94dbb4c8-b2fd-4356-b458-34f860de2057\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5dc3c0-9sdxx" Oct 30 23:57:11.720967 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.720791 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94dbb4c8-b2fd-4356-b458-34f860de2057-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-5dc3c0-9sdxx\" (UID: \"94dbb4c8-b2fd-4356-b458-34f860de2057\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5dc3c0-9sdxx" Oct 30 23:57:11.721449 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.721422 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94dbb4c8-b2fd-4356-b458-34f860de2057-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-5dc3c0-9sdxx\" (UID: \"94dbb4c8-b2fd-4356-b458-34f860de2057\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5dc3c0-9sdxx" Oct 30 23:57:11.723576 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.723550 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94dbb4c8-b2fd-4356-b458-34f860de2057-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-5dc3c0-9sdxx\" (UID: \"94dbb4c8-b2fd-4356-b458-34f860de2057\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5dc3c0-9sdxx" Oct 30 23:57:11.778690 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:11.778663 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9ldc\" (UniqueName: \"kubernetes.io/projected/94dbb4c8-b2fd-4356-b458-34f860de2057-kube-api-access-x9ldc\") pod \"maas-api-revoke-keys-e2e-worker-w5-5dc3c0-9sdxx\" (UID: \"94dbb4c8-b2fd-4356-b458-34f860de2057\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5dc3c0-9sdxx" Oct 30 23:57:12.053421 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:12.053339 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5dc3c0-9sdxx" Oct 30 23:57:12.432046 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:12.432002 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5dc3c0-9sdxx"] Oct 30 23:57:12.433681 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:57:12.433656 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94dbb4c8_b2fd_4356_b458_34f860de2057.slice/crio-43247255dc34a6dc77cdaefe93bf4189f24b4e5fd3b31d60394d5cbc751821c7 WatchSource:0}: Error finding container 43247255dc34a6dc77cdaefe93bf4189f24b4e5fd3b31d60394d5cbc751821c7: Status 404 returned error can't find the container with id 43247255dc34a6dc77cdaefe93bf4189f24b4e5fd3b31d60394d5cbc751821c7 Oct 30 23:57:12.807502 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:12.807466 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5dc3c0-9sdxx" event={"ID":"94dbb4c8-b2fd-4356-b458-34f860de2057","Type":"ContainerStarted","Data":"96ed37579e3eeedb8b302ea720707284b0599dada7c1d5a158737fee400a0403"} Oct 30 23:57:12.807835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:12.807527 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5dc3c0-9sdxx" event={"ID":"94dbb4c8-b2fd-4356-b458-34f860de2057","Type":"ContainerStarted","Data":"43247255dc34a6dc77cdaefe93bf4189f24b4e5fd3b31d60394d5cbc751821c7"} Oct 30 23:57:12.949273 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:12.949220 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5dc3c0-9sdxx" podStartSLOduration=1.9492054730000001 podStartE2EDuration="1.949205473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:12.946593607 +0000 UTC m=+3052.090423831" watchObservedRunningTime="2026-10-30 23:57:12.949205473 +0000 UTC m=+3052.093035713" Oct 30 23:57:13.571204 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:13.571119 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:13.814596 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:13.814558 4952 generic.go:356] "Generic (PLEG): container finished" podID="94dbb4c8-b2fd-4356-b458-34f860de2057" containerID="96ed37579e3eeedb8b302ea720707284b0599dada7c1d5a158737fee400a0403" exitCode=0 Oct 30 23:57:13.815023 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:13.814647 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5dc3c0-9sdxx" event={"ID":"94dbb4c8-b2fd-4356-b458-34f860de2057","Type":"ContainerDied","Data":"96ed37579e3eeedb8b302ea720707284b0599dada7c1d5a158737fee400a0403"} Oct 30 23:57:14.965161 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:14.965118 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5dc3c0-9sdxx" Oct 30 23:57:15.148068 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:15.148021 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:15.150175 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:15.150156 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94dbb4c8-b2fd-4356-b458-34f860de2057-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94dbb4c8-b2fd-4356-b458-34f860de2057-maas-api-xks-ca\") pod \"94dbb4c8-b2fd-4356-b458-34f860de2057\" (UID: \"94dbb4c8-b2fd-4356-b458-34f860de2057\") " Oct 30 23:57:15.150260 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:15.150195 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94dbb4c8-b2fd-4356-b458-34f860de2057-kube-api-access-x9ldc\" (UniqueName: \"kubernetes.io/projected/94dbb4c8-b2fd-4356-b458-34f860de2057-kube-api-access-x9ldc\") pod \"94dbb4c8-b2fd-4356-b458-34f860de2057\" (UID: \"94dbb4c8-b2fd-4356-b458-34f860de2057\") " Oct 30 23:57:15.150260 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:15.150235 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94dbb4c8-b2fd-4356-b458-34f860de2057-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94dbb4c8-b2fd-4356-b458-34f860de2057-maas-api-openshift-ca\") pod \"94dbb4c8-b2fd-4356-b458-34f860de2057\" (UID: \"94dbb4c8-b2fd-4356-b458-34f860de2057\") " Oct 30 23:57:15.150615 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:15.150593 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94dbb4c8-b2fd-4356-b458-34f860de2057-maas-api-openshift-ca" pod "94dbb4c8-b2fd-4356-b458-34f860de2057" (UID: "94dbb4c8-b2fd-4356-b458-34f860de2057"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:57:15.150703 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:15.150686 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94dbb4c8-b2fd-4356-b458-34f860de2057-maas-api-openshift-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:15.152505 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:15.152483 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94dbb4c8-b2fd-4356-b458-34f860de2057-kube-api-access-x9ldc" pod "94dbb4c8-b2fd-4356-b458-34f860de2057" (UID: "94dbb4c8-b2fd-4356-b458-34f860de2057"). InnerVolumeSpecName "kube-api-access-x9ldc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:57:15.152582 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:15.152556 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94dbb4c8-b2fd-4356-b458-34f860de2057-maas-api-xks-ca" pod "94dbb4c8-b2fd-4356-b458-34f860de2057" (UID: "94dbb4c8-b2fd-4356-b458-34f860de2057"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:57:15.251895 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:15.251873 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94dbb4c8-b2fd-4356-b458-34f860de2057-maas-api-xks-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:15.251895 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:15.251894 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9ldc\" (UniqueName: \"kubernetes.io/projected/94dbb4c8-b2fd-4356-b458-34f860de2057-kube-api-access-x9ldc\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:15.403155 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:15.403085 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:15.794725 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:15.794652 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:15.828660 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:15.828629 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5dc3c0-9sdxx" event={"ID":"94dbb4c8-b2fd-4356-b458-34f860de2057","Type":"ContainerDied","Data":"43247255dc34a6dc77cdaefe93bf4189f24b4e5fd3b31d60394d5cbc751821c7"} Oct 30 23:57:15.828660 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:15.828649 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5dc3c0-9sdxx" Oct 30 23:57:15.828925 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:15.828678 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43247255dc34a6dc77cdaefe93bf4189f24b4e5fd3b31d60394d5cbc751821c7" Oct 30 23:57:15.892652 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:15.892628 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj"] Oct 30 23:57:15.941282 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:15.941257 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh"] Oct 30 23:57:16.136902 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:16.136862 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:16.446481 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:16.446416 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:19.152376 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:19.152340 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:20.545753 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:20.545725 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:20.839404 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:20.839369 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:20.847238 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:20.847213 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf"] Oct 30 23:57:20.896796 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:20.896768 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq"] Oct 30 23:57:21.136626 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:21.136549 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:21.142073 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:57:21.142016 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:57:24.229970 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:24.229938 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:24.511134 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:24.511032 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:27.355994 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:27.355963 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:27.828907 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:27.828823 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-aadaa6-85599b78b7-mz9sj"] Oct 30 23:57:27.993433 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:27.993402 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:28.282577 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:28.282547 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:28.586516 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:28.586482 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:30.250013 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:30.249979 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:30.629498 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:30.629465 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:30.912756 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:30.912672 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" podUID="709f007e-0ccc-40f5-816e-ed24702ac107" containerName="main" containerID="cri-o://1e48a65de5154a3b6c7e40cb316d8053b648d517b2451d436ab8cee282a5ad6f" gracePeriod=45 Oct 30 23:57:30.961219 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:30.961184 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" podUID="6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7" containerName="main" containerID="cri-o://67fd5b2ee33c17b620637500c422df0e4e72201dae1572812746117d1ba279ec" gracePeriod=45 Oct 30 23:57:31.131071 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.130248 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:31.220132 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.220106 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" Oct 30 23:57:31.240591 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.240565 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" Oct 30 23:57:31.300421 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.300390 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/709f007e-0ccc-40f5-816e-ed24702ac107-kube-api-access-zxvhm\" (UniqueName: \"kubernetes.io/projected/709f007e-0ccc-40f5-816e-ed24702ac107-kube-api-access-zxvhm\") pod \"709f007e-0ccc-40f5-816e-ed24702ac107\" (UID: \"709f007e-0ccc-40f5-816e-ed24702ac107\") " Oct 30 23:57:31.300865 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.300443 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-home\" (UniqueName: \"kubernetes.io/empty-dir/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-home\") pod \"6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7\" (UID: \"6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7\") " Oct 30 23:57:31.300865 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.300537 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-tmp-dir\") pod \"6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7\" (UID: \"6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7\") " Oct 30 23:57:31.300865 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.300588 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/709f007e-0ccc-40f5-816e-ed24702ac107-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/709f007e-0ccc-40f5-816e-ed24702ac107-model-cache\") pod \"709f007e-0ccc-40f5-816e-ed24702ac107\" (UID: \"709f007e-0ccc-40f5-816e-ed24702ac107\") " Oct 30 23:57:31.300865 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.300612 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-kube-api-access-t2m9c\" (UniqueName: \"kubernetes.io/projected/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-kube-api-access-t2m9c\") pod \"6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7\" (UID: \"6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7\") " Oct 30 23:57:31.300865 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.300646 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-dshm\") pod \"6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7\" (UID: \"6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7\") " Oct 30 23:57:31.300865 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.300673 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-home" pod "6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7" (UID: "6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:31.300865 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.300691 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-model-cache\") pod \"6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7\" (UID: \"6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7\") " Oct 30 23:57:31.300865 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.300741 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/709f007e-0ccc-40f5-816e-ed24702ac107-home\" (UniqueName: \"kubernetes.io/empty-dir/709f007e-0ccc-40f5-816e-ed24702ac107-home\") pod \"709f007e-0ccc-40f5-816e-ed24702ac107\" (UID: \"709f007e-0ccc-40f5-816e-ed24702ac107\") " Oct 30 23:57:31.300865 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.300778 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/709f007e-0ccc-40f5-816e-ed24702ac107-tls-certs\" (UniqueName: \"kubernetes.io/secret/709f007e-0ccc-40f5-816e-ed24702ac107-tls-certs\") pod \"709f007e-0ccc-40f5-816e-ed24702ac107\" (UID: \"709f007e-0ccc-40f5-816e-ed24702ac107\") " Oct 30 23:57:31.300865 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.300831 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/709f007e-0ccc-40f5-816e-ed24702ac107-dshm\" (UniqueName: \"kubernetes.io/empty-dir/709f007e-0ccc-40f5-816e-ed24702ac107-dshm\") pod \"709f007e-0ccc-40f5-816e-ed24702ac107\" (UID: \"709f007e-0ccc-40f5-816e-ed24702ac107\") " Oct 30 23:57:31.300865 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.300819 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-tmp-dir" pod "6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7" (UID: "6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:31.300865 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.300866 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/709f007e-0ccc-40f5-816e-ed24702ac107-model-cache" pod "709f007e-0ccc-40f5-816e-ed24702ac107" (UID: "709f007e-0ccc-40f5-816e-ed24702ac107"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:31.301551 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.301074 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/709f007e-0ccc-40f5-816e-ed24702ac107-home" pod "709f007e-0ccc-40f5-816e-ed24702ac107" (UID: "709f007e-0ccc-40f5-816e-ed24702ac107"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:31.301551 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.301149 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/709f007e-0ccc-40f5-816e-ed24702ac107-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/709f007e-0ccc-40f5-816e-ed24702ac107-tmp-dir\") pod \"709f007e-0ccc-40f5-816e-ed24702ac107\" (UID: \"709f007e-0ccc-40f5-816e-ed24702ac107\") " Oct 30 23:57:31.301551 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.301223 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-tls-certs\") pod \"6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7\" (UID: \"6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7\") " Oct 30 23:57:31.301551 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.301297 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-model-cache" pod "6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7" (UID: "6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:31.301760 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.301606 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-tmp-dir\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:31.301760 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.301625 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/709f007e-0ccc-40f5-816e-ed24702ac107-model-cache\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:31.301760 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.301641 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-model-cache\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:31.301760 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.301655 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/709f007e-0ccc-40f5-816e-ed24702ac107-home\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:31.301760 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.301669 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-home\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:31.301760 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.301731 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/709f007e-0ccc-40f5-816e-ed24702ac107-tmp-dir" pod "709f007e-0ccc-40f5-816e-ed24702ac107" (UID: "709f007e-0ccc-40f5-816e-ed24702ac107"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:31.303604 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.303570 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/709f007e-0ccc-40f5-816e-ed24702ac107-kube-api-access-zxvhm" pod "709f007e-0ccc-40f5-816e-ed24702ac107" (UID: "709f007e-0ccc-40f5-816e-ed24702ac107"). InnerVolumeSpecName "kube-api-access-zxvhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:57:31.303756 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.303691 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-dshm" pod "6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7" (UID: "6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:31.303756 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.303698 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/709f007e-0ccc-40f5-816e-ed24702ac107-dshm" pod "709f007e-0ccc-40f5-816e-ed24702ac107" (UID: "709f007e-0ccc-40f5-816e-ed24702ac107"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:31.304113 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.304092 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/709f007e-0ccc-40f5-816e-ed24702ac107-tls-certs" pod "709f007e-0ccc-40f5-816e-ed24702ac107" (UID: "709f007e-0ccc-40f5-816e-ed24702ac107"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:57:31.304113 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.304094 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-kube-api-access-t2m9c" pod "6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7" (UID: "6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7"). InnerVolumeSpecName "kube-api-access-t2m9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:57:31.304242 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.304123 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-tls-certs" pod "6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7" (UID: "6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:57:31.402964 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.402928 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/709f007e-0ccc-40f5-816e-ed24702ac107-tls-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:31.402964 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.402955 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/709f007e-0ccc-40f5-816e-ed24702ac107-dshm\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:31.402964 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.402965 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/709f007e-0ccc-40f5-816e-ed24702ac107-tmp-dir\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:31.402964 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.402973 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-tls-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:31.403366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.402983 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxvhm\" (UniqueName: \"kubernetes.io/projected/709f007e-0ccc-40f5-816e-ed24702ac107-kube-api-access-zxvhm\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:31.403366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.402993 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2m9c\" (UniqueName: \"kubernetes.io/projected/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-kube-api-access-t2m9c\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:31.403366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.403000 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7-dshm\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:31.932741 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.932695 4952 generic.go:356] "Generic (PLEG): container finished" podID="6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7" containerID="67fd5b2ee33c17b620637500c422df0e4e72201dae1572812746117d1ba279ec" exitCode=0 Oct 30 23:57:31.932860 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.932811 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" Oct 30 23:57:31.932860 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.932824 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" event={"ID":"6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7","Type":"ContainerDied","Data":"67fd5b2ee33c17b620637500c422df0e4e72201dae1572812746117d1ba279ec"} Oct 30 23:57:31.932999 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.932868 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh" event={"ID":"6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7","Type":"ContainerDied","Data":"6fff41e39b21fd6723f5f17bec85564d892f04e84e8699c21f0c84a065243b1c"} Oct 30 23:57:31.932999 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.932890 4952 scope.go:122] "RemoveContainer" containerID="67fd5b2ee33c17b620637500c422df0e4e72201dae1572812746117d1ba279ec" Oct 30 23:57:31.934474 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.934452 4952 generic.go:356] "Generic (PLEG): container finished" podID="709f007e-0ccc-40f5-816e-ed24702ac107" containerID="1e48a65de5154a3b6c7e40cb316d8053b648d517b2451d436ab8cee282a5ad6f" exitCode=0 Oct 30 23:57:31.934550 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.934523 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" Oct 30 23:57:31.934618 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.934537 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" event={"ID":"709f007e-0ccc-40f5-816e-ed24702ac107","Type":"ContainerDied","Data":"1e48a65de5154a3b6c7e40cb316d8053b648d517b2451d436ab8cee282a5ad6f"} Oct 30 23:57:31.934618 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.934582 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj" event={"ID":"709f007e-0ccc-40f5-816e-ed24702ac107","Type":"ContainerDied","Data":"fdc042d8270ac99e7b7f7d5d17ffbc07e4f10512c8f90dd11239d31b5ecb911b"} Oct 30 23:57:31.944582 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.944502 4952 scope.go:122] "RemoveContainer" containerID="67fd5b2ee33c17b620637500c422df0e4e72201dae1572812746117d1ba279ec" Oct 30 23:57:31.944783 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:57:31.944760 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67fd5b2ee33c17b620637500c422df0e4e72201dae1572812746117d1ba279ec\": container with ID starting with 67fd5b2ee33c17b620637500c422df0e4e72201dae1572812746117d1ba279ec not found: ID does not exist" containerID="67fd5b2ee33c17b620637500c422df0e4e72201dae1572812746117d1ba279ec" Oct 30 23:57:31.944880 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.944789 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67fd5b2ee33c17b620637500c422df0e4e72201dae1572812746117d1ba279ec"} err="failed to get container status \"67fd5b2ee33c17b620637500c422df0e4e72201dae1572812746117d1ba279ec\": rpc error: code = NotFound desc = could not find container \"67fd5b2ee33c17b620637500c422df0e4e72201dae1572812746117d1ba279ec\": container with ID starting with 67fd5b2ee33c17b620637500c422df0e4e72201dae1572812746117d1ba279ec not found: ID does not exist" Oct 30 23:57:31.944880 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.944813 4952 scope.go:122] "RemoveContainer" containerID="1e48a65de5154a3b6c7e40cb316d8053b648d517b2451d436ab8cee282a5ad6f" Oct 30 23:57:31.952868 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.952854 4952 scope.go:122] "RemoveContainer" containerID="1e48a65de5154a3b6c7e40cb316d8053b648d517b2451d436ab8cee282a5ad6f" Oct 30 23:57:31.953679 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:57:31.953655 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e48a65de5154a3b6c7e40cb316d8053b648d517b2451d436ab8cee282a5ad6f\": container with ID starting with 1e48a65de5154a3b6c7e40cb316d8053b648d517b2451d436ab8cee282a5ad6f not found: ID does not exist" containerID="1e48a65de5154a3b6c7e40cb316d8053b648d517b2451d436ab8cee282a5ad6f" Oct 30 23:57:31.953760 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:31.953682 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e48a65de5154a3b6c7e40cb316d8053b648d517b2451d436ab8cee282a5ad6f"} err="failed to get container status \"1e48a65de5154a3b6c7e40cb316d8053b648d517b2451d436ab8cee282a5ad6f\": rpc error: code = NotFound desc = could not find container \"1e48a65de5154a3b6c7e40cb316d8053b648d517b2451d436ab8cee282a5ad6f\": container with ID starting with 1e48a65de5154a3b6c7e40cb316d8053b648d517b2451d436ab8cee282a5ad6f not found: ID does not exist" Oct 30 23:57:32.116625 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:32.116588 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj"] Oct 30 23:57:32.122364 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:32.122333 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-5dc3c0/distinct-2-w5-5dc3c0-kserve-67d64ccf47-9rdnj"] Oct 30 23:57:32.276063 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:32.275967 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh"] Oct 30 23:57:32.282479 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:32.282457 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-5dc3c0/premium-w5-5dc3c0-kserve-7b956b9f97-7w2vh"] Oct 30 23:57:32.829738 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:32.829701 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aadaa6-85599b78b7-mz9sj" podUID="9fa21217-0585-4176-8a68-b7ec85bf1068" containerName="payload-pre-processing" containerID="cri-o://87059b69a8f8a54972b0d94c7d42c78146941c1177c2761a33756828198ad328" gracePeriod=25 Oct 30 23:57:32.969626 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:32.969598 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7" path="/var/lib/kubelet/pods/6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7/volumes" Oct 30 23:57:32.970005 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:32.969992 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="709f007e-0ccc-40f5-816e-ed24702ac107" path="/var/lib/kubelet/pods/709f007e-0ccc-40f5-816e-ed24702ac107/volumes" Oct 30 23:57:34.588426 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:34.588389 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:57:35.060056 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.059966 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8ab4cea-mv7np"] Oct 30 23:57:35.060388 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.060375 4952 cpu_manager.go:414] "containerMap: removing container" podUID="94dbb4c8-b2fd-4356-b458-34f860de2057" containerName="revoke-keys" Oct 30 23:57:35.060436 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.060389 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94dbb4c8-b2fd-4356-b458-34f860de2057" containerName="revoke-keys" Oct 30 23:57:35.060436 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.060409 4952 cpu_manager.go:414] "containerMap: removing container" podUID="6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7" containerName="main" Oct 30 23:57:35.060436 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.060414 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7" containerName="main" Oct 30 23:57:35.060532 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.060440 4952 cpu_manager.go:414] "containerMap: removing container" podUID="709f007e-0ccc-40f5-816e-ed24702ac107" containerName="main" Oct 30 23:57:35.060532 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.060446 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="709f007e-0ccc-40f5-816e-ed24702ac107" containerName="main" Oct 30 23:57:35.060532 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.060512 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="709f007e-0ccc-40f5-816e-ed24702ac107" containerName="main" Oct 30 23:57:35.060532 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.060523 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="94dbb4c8-b2fd-4356-b458-34f860de2057" containerName="revoke-keys" Oct 30 23:57:35.060532 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.060530 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c4bf3c4-70de-4d3a-bef9-d23aa84bcfe7" containerName="main" Oct 30 23:57:35.065193 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.065177 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8ab4cea-mv7np" Oct 30 23:57:35.128767 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.128737 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8ab4cea-mv7np"] Oct 30 23:57:35.137958 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.137933 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/314cf6d3-fdc9-4496-b4de-2e531820361c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8ab4cea-mv7np\" (UID: \"314cf6d3-fdc9-4496-b4de-2e531820361c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8ab4cea-mv7np" Oct 30 23:57:35.138091 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.138003 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/314cf6d3-fdc9-4496-b4de-2e531820361c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8ab4cea-mv7np\" (UID: \"314cf6d3-fdc9-4496-b4de-2e531820361c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8ab4cea-mv7np" Oct 30 23:57:35.138174 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.138113 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsz5z\" (UniqueName: \"kubernetes.io/projected/314cf6d3-fdc9-4496-b4de-2e531820361c-kube-api-access-bsz5z\") pod \"maas-api-revoke-sub-a8ab4cea-mv7np\" (UID: \"314cf6d3-fdc9-4496-b4de-2e531820361c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8ab4cea-mv7np" Oct 30 23:57:35.175269 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.175248 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:57:35.175376 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.175312 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\"" Oct 30 23:57:35.238828 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.238799 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/314cf6d3-fdc9-4496-b4de-2e531820361c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8ab4cea-mv7np\" (UID: \"314cf6d3-fdc9-4496-b4de-2e531820361c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8ab4cea-mv7np" Oct 30 23:57:35.238948 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.238894 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/314cf6d3-fdc9-4496-b4de-2e531820361c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8ab4cea-mv7np\" (UID: \"314cf6d3-fdc9-4496-b4de-2e531820361c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8ab4cea-mv7np" Oct 30 23:57:35.238948 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.238935 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsz5z\" (UniqueName: \"kubernetes.io/projected/314cf6d3-fdc9-4496-b4de-2e531820361c-kube-api-access-bsz5z\") pod \"maas-api-revoke-sub-a8ab4cea-mv7np\" (UID: \"314cf6d3-fdc9-4496-b4de-2e531820361c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8ab4cea-mv7np" Oct 30 23:57:35.239560 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.239534 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/314cf6d3-fdc9-4496-b4de-2e531820361c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8ab4cea-mv7np\" (UID: \"314cf6d3-fdc9-4496-b4de-2e531820361c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8ab4cea-mv7np" Oct 30 23:57:35.241857 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.241837 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/314cf6d3-fdc9-4496-b4de-2e531820361c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8ab4cea-mv7np\" (UID: \"314cf6d3-fdc9-4496-b4de-2e531820361c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8ab4cea-mv7np" Oct 30 23:57:35.298913 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.298882 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsz5z\" (UniqueName: \"kubernetes.io/projected/314cf6d3-fdc9-4496-b4de-2e531820361c-kube-api-access-bsz5z\") pod \"maas-api-revoke-sub-a8ab4cea-mv7np\" (UID: \"314cf6d3-fdc9-4496-b4de-2e531820361c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8ab4cea-mv7np" Oct 30 23:57:35.313527 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.313463 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee80d40a-vszm6"] Oct 30 23:57:35.317662 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.317641 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee80d40a-vszm6" Oct 30 23:57:35.376549 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.376524 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee80d40a-vszm6"] Oct 30 23:57:35.381447 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.381423 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8ab4cea-mv7np" Oct 30 23:57:35.443713 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.443673 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789348bc-4698-479b-b2b0-da808059305f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee80d40a-vszm6\" (UID: \"789348bc-4698-479b-b2b0-da808059305f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee80d40a-vszm6" Oct 30 23:57:35.443857 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.443755 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789348bc-4698-479b-b2b0-da808059305f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee80d40a-vszm6\" (UID: \"789348bc-4698-479b-b2b0-da808059305f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee80d40a-vszm6" Oct 30 23:57:35.443931 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.443904 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68qxf\" (UniqueName: \"kubernetes.io/projected/789348bc-4698-479b-b2b0-da808059305f-kube-api-access-68qxf\") pod \"maas-api-revoke-sub-ee80d40a-vszm6\" (UID: \"789348bc-4698-479b-b2b0-da808059305f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee80d40a-vszm6" Oct 30 23:57:35.544905 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.544870 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789348bc-4698-479b-b2b0-da808059305f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee80d40a-vszm6\" (UID: \"789348bc-4698-479b-b2b0-da808059305f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee80d40a-vszm6" Oct 30 23:57:35.545096 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.544965 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68qxf\" (UniqueName: \"kubernetes.io/projected/789348bc-4698-479b-b2b0-da808059305f-kube-api-access-68qxf\") pod \"maas-api-revoke-sub-ee80d40a-vszm6\" (UID: \"789348bc-4698-479b-b2b0-da808059305f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee80d40a-vszm6" Oct 30 23:57:35.545096 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.544994 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789348bc-4698-479b-b2b0-da808059305f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee80d40a-vszm6\" (UID: \"789348bc-4698-479b-b2b0-da808059305f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee80d40a-vszm6" Oct 30 23:57:35.545606 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.545582 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789348bc-4698-479b-b2b0-da808059305f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee80d40a-vszm6\" (UID: \"789348bc-4698-479b-b2b0-da808059305f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee80d40a-vszm6" Oct 30 23:57:35.547515 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.547488 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789348bc-4698-479b-b2b0-da808059305f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee80d40a-vszm6\" (UID: \"789348bc-4698-479b-b2b0-da808059305f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee80d40a-vszm6" Oct 30 23:57:35.602685 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.602662 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68qxf\" (UniqueName: \"kubernetes.io/projected/789348bc-4698-479b-b2b0-da808059305f-kube-api-access-68qxf\") pod \"maas-api-revoke-sub-ee80d40a-vszm6\" (UID: \"789348bc-4698-479b-b2b0-da808059305f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee80d40a-vszm6" Oct 30 23:57:35.631510 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.631489 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee80d40a-vszm6" Oct 30 23:57:35.769261 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.769237 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8ab4cea-mv7np"] Oct 30 23:57:35.770942 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:57:35.770913 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod314cf6d3_fdc9_4496_b4de_2e531820361c.slice/crio-df41723b816490b306c1a3ef87d678ddc7406458e9638b6352d21d7c42e2b23b WatchSource:0}: Error finding container df41723b816490b306c1a3ef87d678ddc7406458e9638b6352d21d7c42e2b23b: Status 404 returned error can't find the container with id df41723b816490b306c1a3ef87d678ddc7406458e9638b6352d21d7c42e2b23b Oct 30 23:57:35.869301 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.869205 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" podUID="959b92af-9035-4dbd-aed3-5db4daaa63a8" containerName="main" containerID="cri-o://9f7156db0d849717653d4f2107439baedc5f52f49bd3109c08037709775c77c4" gracePeriod=45 Oct 30 23:57:35.916679 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.916644 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" podUID="145e7c01-efec-46e6-a748-56b39de76f83" containerName="main" containerID="cri-o://0b7590c5d1b17285ac8a1ada673d7c46d3a6ec238a29c80a0518e93bb4e35798" gracePeriod=45 Oct 30 23:57:35.962024 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:35.961998 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8ab4cea-mv7np" event={"ID":"314cf6d3-fdc9-4496-b4de-2e531820361c","Type":"ContainerStarted","Data":"df41723b816490b306c1a3ef87d678ddc7406458e9638b6352d21d7c42e2b23b"} Oct 30 23:57:36.007053 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.007016 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee80d40a-vszm6"] Oct 30 23:57:36.007836 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:57:36.007810 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod789348bc_4698_479b_b2b0_da808059305f.slice/crio-90dc95c783ad1e93b2cb8bfe233af1c05693e6d1a5d36025c87fe84b6757123e WatchSource:0}: Error finding container 90dc95c783ad1e93b2cb8bfe233af1c05693e6d1a5d36025c87fe84b6757123e: Status 404 returned error can't find the container with id 90dc95c783ad1e93b2cb8bfe233af1c05693e6d1a5d36025c87fe84b6757123e Oct 30 23:57:36.164613 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.164588 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" Oct 30 23:57:36.202387 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.202366 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" Oct 30 23:57:36.252431 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.252400 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/145e7c01-efec-46e6-a748-56b39de76f83-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/145e7c01-efec-46e6-a748-56b39de76f83-tmp-dir\") pod \"145e7c01-efec-46e6-a748-56b39de76f83\" (UID: \"145e7c01-efec-46e6-a748-56b39de76f83\") " Oct 30 23:57:36.252601 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.252490 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/959b92af-9035-4dbd-aed3-5db4daaa63a8-kube-api-access-tsqpr\" (UniqueName: \"kubernetes.io/projected/959b92af-9035-4dbd-aed3-5db4daaa63a8-kube-api-access-tsqpr\") pod \"959b92af-9035-4dbd-aed3-5db4daaa63a8\" (UID: \"959b92af-9035-4dbd-aed3-5db4daaa63a8\") " Oct 30 23:57:36.252601 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.252531 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/145e7c01-efec-46e6-a748-56b39de76f83-dshm\" (UniqueName: \"kubernetes.io/empty-dir/145e7c01-efec-46e6-a748-56b39de76f83-dshm\") pod \"145e7c01-efec-46e6-a748-56b39de76f83\" (UID: \"145e7c01-efec-46e6-a748-56b39de76f83\") " Oct 30 23:57:36.252601 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.252561 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/145e7c01-efec-46e6-a748-56b39de76f83-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/145e7c01-efec-46e6-a748-56b39de76f83-model-cache\") pod \"145e7c01-efec-46e6-a748-56b39de76f83\" (UID: \"145e7c01-efec-46e6-a748-56b39de76f83\") " Oct 30 23:57:36.252601 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.252592 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/145e7c01-efec-46e6-a748-56b39de76f83-home\" (UniqueName: \"kubernetes.io/empty-dir/145e7c01-efec-46e6-a748-56b39de76f83-home\") pod \"145e7c01-efec-46e6-a748-56b39de76f83\" (UID: \"145e7c01-efec-46e6-a748-56b39de76f83\") " Oct 30 23:57:36.252820 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.252637 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/145e7c01-efec-46e6-a748-56b39de76f83-tls-certs\" (UniqueName: \"kubernetes.io/secret/145e7c01-efec-46e6-a748-56b39de76f83-tls-certs\") pod \"145e7c01-efec-46e6-a748-56b39de76f83\" (UID: \"145e7c01-efec-46e6-a748-56b39de76f83\") " Oct 30 23:57:36.252820 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.252693 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/959b92af-9035-4dbd-aed3-5db4daaa63a8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/959b92af-9035-4dbd-aed3-5db4daaa63a8-tmp-dir\") pod \"959b92af-9035-4dbd-aed3-5db4daaa63a8\" (UID: \"959b92af-9035-4dbd-aed3-5db4daaa63a8\") " Oct 30 23:57:36.252820 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.252723 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/145e7c01-efec-46e6-a748-56b39de76f83-kube-api-access-nmtvh\" (UniqueName: \"kubernetes.io/projected/145e7c01-efec-46e6-a748-56b39de76f83-kube-api-access-nmtvh\") pod \"145e7c01-efec-46e6-a748-56b39de76f83\" (UID: \"145e7c01-efec-46e6-a748-56b39de76f83\") " Oct 30 23:57:36.252820 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.252815 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/959b92af-9035-4dbd-aed3-5db4daaa63a8-tls-certs\" (UniqueName: \"kubernetes.io/secret/959b92af-9035-4dbd-aed3-5db4daaa63a8-tls-certs\") pod \"959b92af-9035-4dbd-aed3-5db4daaa63a8\" (UID: \"959b92af-9035-4dbd-aed3-5db4daaa63a8\") " Oct 30 23:57:36.253027 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.252847 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/959b92af-9035-4dbd-aed3-5db4daaa63a8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/959b92af-9035-4dbd-aed3-5db4daaa63a8-dshm\") pod \"959b92af-9035-4dbd-aed3-5db4daaa63a8\" (UID: \"959b92af-9035-4dbd-aed3-5db4daaa63a8\") " Oct 30 23:57:36.253027 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.252886 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/959b92af-9035-4dbd-aed3-5db4daaa63a8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/959b92af-9035-4dbd-aed3-5db4daaa63a8-model-cache\") pod \"959b92af-9035-4dbd-aed3-5db4daaa63a8\" (UID: \"959b92af-9035-4dbd-aed3-5db4daaa63a8\") " Oct 30 23:57:36.253027 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.252929 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/959b92af-9035-4dbd-aed3-5db4daaa63a8-home\" (UniqueName: \"kubernetes.io/empty-dir/959b92af-9035-4dbd-aed3-5db4daaa63a8-home\") pod \"959b92af-9035-4dbd-aed3-5db4daaa63a8\" (UID: \"959b92af-9035-4dbd-aed3-5db4daaa63a8\") " Oct 30 23:57:36.253536 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.253511 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/959b92af-9035-4dbd-aed3-5db4daaa63a8-home" pod "959b92af-9035-4dbd-aed3-5db4daaa63a8" (UID: "959b92af-9035-4dbd-aed3-5db4daaa63a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:36.254099 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.254004 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/959b92af-9035-4dbd-aed3-5db4daaa63a8-tmp-dir" pod "959b92af-9035-4dbd-aed3-5db4daaa63a8" (UID: "959b92af-9035-4dbd-aed3-5db4daaa63a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:36.254737 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.254311 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/145e7c01-efec-46e6-a748-56b39de76f83-home" pod "145e7c01-efec-46e6-a748-56b39de76f83" (UID: "145e7c01-efec-46e6-a748-56b39de76f83"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:36.254737 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.254678 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/959b92af-9035-4dbd-aed3-5db4daaa63a8-model-cache" pod "959b92af-9035-4dbd-aed3-5db4daaa63a8" (UID: "959b92af-9035-4dbd-aed3-5db4daaa63a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:36.254737 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.254700 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/145e7c01-efec-46e6-a748-56b39de76f83-model-cache" pod "145e7c01-efec-46e6-a748-56b39de76f83" (UID: "145e7c01-efec-46e6-a748-56b39de76f83"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:36.255069 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.255016 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/145e7c01-efec-46e6-a748-56b39de76f83-tmp-dir" pod "145e7c01-efec-46e6-a748-56b39de76f83" (UID: "145e7c01-efec-46e6-a748-56b39de76f83"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:36.257482 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.257440 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/959b92af-9035-4dbd-aed3-5db4daaa63a8-tls-certs" pod "959b92af-9035-4dbd-aed3-5db4daaa63a8" (UID: "959b92af-9035-4dbd-aed3-5db4daaa63a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:57:36.257482 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.257469 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/959b92af-9035-4dbd-aed3-5db4daaa63a8-dshm" pod "959b92af-9035-4dbd-aed3-5db4daaa63a8" (UID: "959b92af-9035-4dbd-aed3-5db4daaa63a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:36.259163 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.259097 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/145e7c01-efec-46e6-a748-56b39de76f83-kube-api-access-nmtvh" pod "145e7c01-efec-46e6-a748-56b39de76f83" (UID: "145e7c01-efec-46e6-a748-56b39de76f83"). InnerVolumeSpecName "kube-api-access-nmtvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:57:36.259412 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.259380 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/959b92af-9035-4dbd-aed3-5db4daaa63a8-kube-api-access-tsqpr" pod "959b92af-9035-4dbd-aed3-5db4daaa63a8" (UID: "959b92af-9035-4dbd-aed3-5db4daaa63a8"). InnerVolumeSpecName "kube-api-access-tsqpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:57:36.260381 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.260361 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/145e7c01-efec-46e6-a748-56b39de76f83-tls-certs" pod "145e7c01-efec-46e6-a748-56b39de76f83" (UID: "145e7c01-efec-46e6-a748-56b39de76f83"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:57:36.260752 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.260726 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/145e7c01-efec-46e6-a748-56b39de76f83-dshm" pod "145e7c01-efec-46e6-a748-56b39de76f83" (UID: "145e7c01-efec-46e6-a748-56b39de76f83"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:36.312380 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.312347 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:36.354456 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.354433 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/959b92af-9035-4dbd-aed3-5db4daaa63a8-tls-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:36.354456 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.354454 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/959b92af-9035-4dbd-aed3-5db4daaa63a8-dshm\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:36.354625 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.354464 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/959b92af-9035-4dbd-aed3-5db4daaa63a8-model-cache\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:36.354625 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.354472 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/959b92af-9035-4dbd-aed3-5db4daaa63a8-home\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:36.354625 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.354482 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/145e7c01-efec-46e6-a748-56b39de76f83-tmp-dir\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:36.354625 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.354490 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsqpr\" (UniqueName: \"kubernetes.io/projected/959b92af-9035-4dbd-aed3-5db4daaa63a8-kube-api-access-tsqpr\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:36.354625 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.354499 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/145e7c01-efec-46e6-a748-56b39de76f83-dshm\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:36.354625 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.354507 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/145e7c01-efec-46e6-a748-56b39de76f83-model-cache\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:36.354625 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.354514 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/145e7c01-efec-46e6-a748-56b39de76f83-home\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:36.354625 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.354522 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/145e7c01-efec-46e6-a748-56b39de76f83-tls-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:36.354625 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.354529 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/959b92af-9035-4dbd-aed3-5db4daaa63a8-tmp-dir\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:36.354625 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.354539 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmtvh\" (UniqueName: \"kubernetes.io/projected/145e7c01-efec-46e6-a748-56b39de76f83-kube-api-access-nmtvh\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:36.969931 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.969897 4952 generic.go:356] "Generic (PLEG): container finished" podID="959b92af-9035-4dbd-aed3-5db4daaa63a8" containerID="9f7156db0d849717653d4f2107439baedc5f52f49bd3109c08037709775c77c4" exitCode=0 Oct 30 23:57:36.970418 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.970025 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" Oct 30 23:57:36.971710 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.971683 4952 generic.go:356] "Generic (PLEG): container finished" podID="145e7c01-efec-46e6-a748-56b39de76f83" containerID="0b7590c5d1b17285ac8a1ada673d7c46d3a6ec238a29c80a0518e93bb4e35798" exitCode=0 Oct 30 23:57:36.971853 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.971787 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" Oct 30 23:57:36.971917 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.971869 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" event={"ID":"959b92af-9035-4dbd-aed3-5db4daaa63a8","Type":"ContainerDied","Data":"9f7156db0d849717653d4f2107439baedc5f52f49bd3109c08037709775c77c4"} Oct 30 23:57:36.971917 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.971907 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf" event={"ID":"959b92af-9035-4dbd-aed3-5db4daaa63a8","Type":"ContainerDied","Data":"6e4f9222dfdde8e0e77223460ad74bebbdb111282c597847db3a7d052e05d0ed"} Oct 30 23:57:36.972028 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.971919 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" event={"ID":"145e7c01-efec-46e6-a748-56b39de76f83","Type":"ContainerDied","Data":"0b7590c5d1b17285ac8a1ada673d7c46d3a6ec238a29c80a0518e93bb4e35798"} Oct 30 23:57:36.972028 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.971931 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq" event={"ID":"145e7c01-efec-46e6-a748-56b39de76f83","Type":"ContainerDied","Data":"85c2798843abd3594b37119e9188654f94f1bf939b4a34862fa62d25d63d0b37"} Oct 30 23:57:36.972028 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.971947 4952 scope.go:122] "RemoveContainer" containerID="9f7156db0d849717653d4f2107439baedc5f52f49bd3109c08037709775c77c4" Oct 30 23:57:36.974416 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.974393 4952 generic.go:356] "Generic (PLEG): container finished" podID="314cf6d3-fdc9-4496-b4de-2e531820361c" containerID="02cb6003ac30174ac5d59f0161caa5f8bb5e8235c46d44f21db98b4406ee6b52" exitCode=0 Oct 30 23:57:36.974519 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.974485 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8ab4cea-mv7np" event={"ID":"314cf6d3-fdc9-4496-b4de-2e531820361c","Type":"ContainerDied","Data":"02cb6003ac30174ac5d59f0161caa5f8bb5e8235c46d44f21db98b4406ee6b52"} Oct 30 23:57:36.981186 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.981104 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee80d40a-vszm6" event={"ID":"789348bc-4698-479b-b2b0-da808059305f","Type":"ContainerStarted","Data":"6c4492230600cb02e2b3c38b8ff473851e2ce84687eb9f3d1c85c5eaa2edb756"} Oct 30 23:57:36.981622 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.981303 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee80d40a-vszm6" event={"ID":"789348bc-4698-479b-b2b0-da808059305f","Type":"ContainerStarted","Data":"90dc95c783ad1e93b2cb8bfe233af1c05693e6d1a5d36025c87fe84b6757123e"} Oct 30 23:57:36.985358 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.985340 4952 scope.go:122] "RemoveContainer" containerID="9f7156db0d849717653d4f2107439baedc5f52f49bd3109c08037709775c77c4" Oct 30 23:57:36.985672 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:57:36.985641 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f7156db0d849717653d4f2107439baedc5f52f49bd3109c08037709775c77c4\": container with ID starting with 9f7156db0d849717653d4f2107439baedc5f52f49bd3109c08037709775c77c4 not found: ID does not exist" containerID="9f7156db0d849717653d4f2107439baedc5f52f49bd3109c08037709775c77c4" Oct 30 23:57:36.985765 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.985683 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f7156db0d849717653d4f2107439baedc5f52f49bd3109c08037709775c77c4"} err="failed to get container status \"9f7156db0d849717653d4f2107439baedc5f52f49bd3109c08037709775c77c4\": rpc error: code = NotFound desc = could not find container \"9f7156db0d849717653d4f2107439baedc5f52f49bd3109c08037709775c77c4\": container with ID starting with 9f7156db0d849717653d4f2107439baedc5f52f49bd3109c08037709775c77c4 not found: ID does not exist" Oct 30 23:57:36.985765 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.985706 4952 scope.go:122] "RemoveContainer" containerID="0b7590c5d1b17285ac8a1ada673d7c46d3a6ec238a29c80a0518e93bb4e35798" Oct 30 23:57:36.995597 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.995576 4952 scope.go:122] "RemoveContainer" containerID="0b7590c5d1b17285ac8a1ada673d7c46d3a6ec238a29c80a0518e93bb4e35798" Oct 30 23:57:36.995834 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:57:36.995817 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b7590c5d1b17285ac8a1ada673d7c46d3a6ec238a29c80a0518e93bb4e35798\": container with ID starting with 0b7590c5d1b17285ac8a1ada673d7c46d3a6ec238a29c80a0518e93bb4e35798 not found: ID does not exist" containerID="0b7590c5d1b17285ac8a1ada673d7c46d3a6ec238a29c80a0518e93bb4e35798" Oct 30 23:57:36.995875 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:36.995842 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b7590c5d1b17285ac8a1ada673d7c46d3a6ec238a29c80a0518e93bb4e35798"} err="failed to get container status \"0b7590c5d1b17285ac8a1ada673d7c46d3a6ec238a29c80a0518e93bb4e35798\": rpc error: code = NotFound desc = could not find container \"0b7590c5d1b17285ac8a1ada673d7c46d3a6ec238a29c80a0518e93bb4e35798\": container with ID starting with 0b7590c5d1b17285ac8a1ada673d7c46d3a6ec238a29c80a0518e93bb4e35798 not found: ID does not exist" Oct 30 23:57:37.148575 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:37.148530 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee80d40a-vszm6" podStartSLOduration=2.148517258 podStartE2EDuration="2.148517258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:37.145893284 +0000 UTC m=+3076.289723490" watchObservedRunningTime="2026-10-30 23:57:37.148517258 +0000 UTC m=+3076.292347461" Oct 30 23:57:37.261929 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:37.261900 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf"] Oct 30 23:57:37.265148 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:37.265102 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-aadaa6/premium-w6-aadaa6-kserve-8586b84766-zddpf"] Oct 30 23:57:37.494456 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:37.494355 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq"] Oct 30 23:57:37.496756 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:37.496729 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-aadaa6/simulator-w6-aadaa6-kserve-9795784cf-skbvq"] Oct 30 23:57:37.988021 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:37.987987 4952 generic.go:356] "Generic (PLEG): container finished" podID="789348bc-4698-479b-b2b0-da808059305f" containerID="6c4492230600cb02e2b3c38b8ff473851e2ce84687eb9f3d1c85c5eaa2edb756" exitCode=0 Oct 30 23:57:37.988419 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:37.988071 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee80d40a-vszm6" event={"ID":"789348bc-4698-479b-b2b0-da808059305f","Type":"ContainerDied","Data":"6c4492230600cb02e2b3c38b8ff473851e2ce84687eb9f3d1c85c5eaa2edb756"} Oct 30 23:57:38.132565 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:38.132539 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8ab4cea-mv7np" Oct 30 23:57:38.270189 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:38.270112 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/314cf6d3-fdc9-4496-b4de-2e531820361c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/314cf6d3-fdc9-4496-b4de-2e531820361c-maas-api-xks-ca\") pod \"314cf6d3-fdc9-4496-b4de-2e531820361c\" (UID: \"314cf6d3-fdc9-4496-b4de-2e531820361c\") " Oct 30 23:57:38.270358 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:38.270200 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/314cf6d3-fdc9-4496-b4de-2e531820361c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/314cf6d3-fdc9-4496-b4de-2e531820361c-maas-api-openshift-ca\") pod \"314cf6d3-fdc9-4496-b4de-2e531820361c\" (UID: \"314cf6d3-fdc9-4496-b4de-2e531820361c\") " Oct 30 23:57:38.270358 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:38.270352 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/314cf6d3-fdc9-4496-b4de-2e531820361c-kube-api-access-bsz5z\" (UniqueName: \"kubernetes.io/projected/314cf6d3-fdc9-4496-b4de-2e531820361c-kube-api-access-bsz5z\") pod \"314cf6d3-fdc9-4496-b4de-2e531820361c\" (UID: \"314cf6d3-fdc9-4496-b4de-2e531820361c\") " Oct 30 23:57:38.270558 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:38.270532 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/314cf6d3-fdc9-4496-b4de-2e531820361c-maas-api-openshift-ca" pod "314cf6d3-fdc9-4496-b4de-2e531820361c" (UID: "314cf6d3-fdc9-4496-b4de-2e531820361c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:57:38.270725 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:38.270708 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/314cf6d3-fdc9-4496-b4de-2e531820361c-maas-api-openshift-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:38.272394 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:38.272364 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/314cf6d3-fdc9-4496-b4de-2e531820361c-maas-api-xks-ca" pod "314cf6d3-fdc9-4496-b4de-2e531820361c" (UID: "314cf6d3-fdc9-4496-b4de-2e531820361c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:57:38.272495 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:38.272419 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/314cf6d3-fdc9-4496-b4de-2e531820361c-kube-api-access-bsz5z" pod "314cf6d3-fdc9-4496-b4de-2e531820361c" (UID: "314cf6d3-fdc9-4496-b4de-2e531820361c"). InnerVolumeSpecName "kube-api-access-bsz5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:57:38.371535 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:38.371504 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsz5z\" (UniqueName: \"kubernetes.io/projected/314cf6d3-fdc9-4496-b4de-2e531820361c-kube-api-access-bsz5z\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:38.371535 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:38.371530 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/314cf6d3-fdc9-4496-b4de-2e531820361c-maas-api-xks-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:38.423320 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:38.423291 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:38.675841 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:38.675805 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:38.971366 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:38.971296 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="145e7c01-efec-46e6-a748-56b39de76f83" path="/var/lib/kubelet/pods/145e7c01-efec-46e6-a748-56b39de76f83/volumes" Oct 30 23:57:38.971711 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:38.971697 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="959b92af-9035-4dbd-aed3-5db4daaa63a8" path="/var/lib/kubelet/pods/959b92af-9035-4dbd-aed3-5db4daaa63a8/volumes" Oct 30 23:57:38.996534 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:38.996511 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8ab4cea-mv7np" Oct 30 23:57:38.996822 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:38.996542 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8ab4cea-mv7np" event={"ID":"314cf6d3-fdc9-4496-b4de-2e531820361c","Type":"ContainerDied","Data":"df41723b816490b306c1a3ef87d678ddc7406458e9638b6352d21d7c42e2b23b"} Oct 30 23:57:38.996822 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:38.996580 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df41723b816490b306c1a3ef87d678ddc7406458e9638b6352d21d7c42e2b23b" Oct 30 23:57:39.069012 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:39.068987 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:39.145385 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:39.145362 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee80d40a-vszm6" Oct 30 23:57:39.281320 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:39.281257 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/789348bc-4698-479b-b2b0-da808059305f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789348bc-4698-479b-b2b0-da808059305f-maas-api-openshift-ca\") pod \"789348bc-4698-479b-b2b0-da808059305f\" (UID: \"789348bc-4698-479b-b2b0-da808059305f\") " Oct 30 23:57:39.281457 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:39.281318 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/789348bc-4698-479b-b2b0-da808059305f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789348bc-4698-479b-b2b0-da808059305f-maas-api-xks-ca\") pod \"789348bc-4698-479b-b2b0-da808059305f\" (UID: \"789348bc-4698-479b-b2b0-da808059305f\") " Oct 30 23:57:39.281457 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:39.281389 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/789348bc-4698-479b-b2b0-da808059305f-kube-api-access-68qxf\" (UniqueName: \"kubernetes.io/projected/789348bc-4698-479b-b2b0-da808059305f-kube-api-access-68qxf\") pod \"789348bc-4698-479b-b2b0-da808059305f\" (UID: \"789348bc-4698-479b-b2b0-da808059305f\") " Oct 30 23:57:39.281581 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:39.281562 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789348bc-4698-479b-b2b0-da808059305f-maas-api-openshift-ca" pod "789348bc-4698-479b-b2b0-da808059305f" (UID: "789348bc-4698-479b-b2b0-da808059305f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:57:39.281747 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:39.281725 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789348bc-4698-479b-b2b0-da808059305f-maas-api-openshift-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:39.283583 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:39.283556 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/789348bc-4698-479b-b2b0-da808059305f-maas-api-xks-ca" pod "789348bc-4698-479b-b2b0-da808059305f" (UID: "789348bc-4698-479b-b2b0-da808059305f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:57:39.283907 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:39.283887 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789348bc-4698-479b-b2b0-da808059305f-kube-api-access-68qxf" pod "789348bc-4698-479b-b2b0-da808059305f" (UID: "789348bc-4698-479b-b2b0-da808059305f"). InnerVolumeSpecName "kube-api-access-68qxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:57:39.342101 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:39.342077 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:39.382554 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:39.382531 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789348bc-4698-479b-b2b0-da808059305f-maas-api-xks-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:39.382648 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:39.382559 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68qxf\" (UniqueName: \"kubernetes.io/projected/789348bc-4698-479b-b2b0-da808059305f-kube-api-access-68qxf\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:40.004432 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:40.004405 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee80d40a-vszm6" Oct 30 23:57:40.004432 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:40.004413 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee80d40a-vszm6" event={"ID":"789348bc-4698-479b-b2b0-da808059305f","Type":"ContainerDied","Data":"90dc95c783ad1e93b2cb8bfe233af1c05693e6d1a5d36025c87fe84b6757123e"} Oct 30 23:57:40.004857 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:40.004451 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90dc95c783ad1e93b2cb8bfe233af1c05693e6d1a5d36025c87fe84b6757123e" Oct 30 23:57:41.357459 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:41.357421 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aadaa6-85599b78b7-mz9sj" podUID="9fa21217-0585-4176-8a68-b7ec85bf1068" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 23:57:41.498749 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:41.498719 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:44.131324 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:44.131289 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj"] Oct 30 23:57:44.181198 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:44.181166 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb"] Oct 30 23:57:44.397825 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:44.397745 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:46.961133 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:46.961082 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:50.888160 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:50.888123 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:57:51.141955 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:57:51.141866 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:57:51.207022 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:51.200737 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b81a7a-78746cfd9c-ql62c"] Oct 30 23:57:51.320961 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:51.320926 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b81a7a-5d9fb6cfb7-6wttp"] Oct 30 23:57:51.357942 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:51.357905 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aadaa6-85599b78b7-mz9sj" podUID="9fa21217-0585-4176-8a68-b7ec85bf1068" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 23:57:51.417781 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:51.417690 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:51.668019 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:51.667941 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:51.906413 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:51.906373 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:52.589062 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:52.589010 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:53.018071 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:53.017991 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:53.091402 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:53.091327 4952 generic.go:356] "Generic (PLEG): container finished" podID="9fa21217-0585-4176-8a68-b7ec85bf1068" containerID="87059b69a8f8a54972b0d94c7d42c78146941c1177c2761a33756828198ad328" exitCode=0 Oct 30 23:57:53.091871 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:53.091791 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aadaa6-85599b78b7-mz9sj" event={"ID":"9fa21217-0585-4176-8a68-b7ec85bf1068","Type":"ContainerDied","Data":"87059b69a8f8a54972b0d94c7d42c78146941c1177c2761a33756828198ad328"} Oct 30 23:57:53.130694 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:53.129679 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aadaa6-85599b78b7-mz9sj" Oct 30 23:57:53.138769 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:53.138737 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fa21217-0585-4176-8a68-b7ec85bf1068-kube-api-access-x56ql\" (UniqueName: \"kubernetes.io/projected/9fa21217-0585-4176-8a68-b7ec85bf1068-kube-api-access-x56ql\") pod \"9fa21217-0585-4176-8a68-b7ec85bf1068\" (UID: \"9fa21217-0585-4176-8a68-b7ec85bf1068\") " Oct 30 23:57:53.138892 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:53.138827 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fa21217-0585-4176-8a68-b7ec85bf1068-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fa21217-0585-4176-8a68-b7ec85bf1068-plugins-config-volume\") pod \"9fa21217-0585-4176-8a68-b7ec85bf1068\" (UID: \"9fa21217-0585-4176-8a68-b7ec85bf1068\") " Oct 30 23:57:53.139642 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:53.139619 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fa21217-0585-4176-8a68-b7ec85bf1068-plugins-config-volume" pod "9fa21217-0585-4176-8a68-b7ec85bf1068" (UID: "9fa21217-0585-4176-8a68-b7ec85bf1068"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:57:53.144464 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:53.144422 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fa21217-0585-4176-8a68-b7ec85bf1068-kube-api-access-x56ql" pod "9fa21217-0585-4176-8a68-b7ec85bf1068" (UID: "9fa21217-0585-4176-8a68-b7ec85bf1068"). InnerVolumeSpecName "kube-api-access-x56ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:57:53.240314 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:53.240250 4952 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fa21217-0585-4176-8a68-b7ec85bf1068-plugins-config-volume\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:53.240314 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:53.240286 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x56ql\" (UniqueName: \"kubernetes.io/projected/9fa21217-0585-4176-8a68-b7ec85bf1068-kube-api-access-x56ql\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:53.326373 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:53.326331 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:53.535231 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:53.535153 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5dc3c0-9sdxx"] Oct 30 23:57:53.545393 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:53.545355 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5dc3c0-9sdxx"] Oct 30 23:57:54.058203 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:54.058168 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:54.100097 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:54.100062 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aadaa6-85599b78b7-mz9sj" event={"ID":"9fa21217-0585-4176-8a68-b7ec85bf1068","Type":"ContainerDied","Data":"3dc59c3c9588b9e5f72459fe2588b07d2f41bcec05ba18e7749b004e163a66e3"} Oct 30 23:57:54.100097 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:54.100030 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-aadaa6-85599b78b7-mz9sj" Oct 30 23:57:54.100297 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:54.100129 4952 scope.go:122] "RemoveContainer" containerID="87059b69a8f8a54972b0d94c7d42c78146941c1177c2761a33756828198ad328" Oct 30 23:57:54.272880 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:54.272842 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-aadaa6-85599b78b7-mz9sj"] Oct 30 23:57:54.276657 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:54.276630 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-aadaa6-85599b78b7-mz9sj"] Oct 30 23:57:54.297943 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:54.297906 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:54.537803 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:54.537699 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:54.792680 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:54.792593 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:54.969967 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:54.969933 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94dbb4c8-b2fd-4356-b458-34f860de2057" path="/var/lib/kubelet/pods/94dbb4c8-b2fd-4356-b458-34f860de2057/volumes" Oct 30 23:57:54.970344 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:54.970330 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fa21217-0585-4176-8a68-b7ec85bf1068" path="/var/lib/kubelet/pods/9fa21217-0585-4176-8a68-b7ec85bf1068/volumes" Oct 30 23:57:55.284752 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:55.284717 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:55.545221 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:55.545119 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:56.155778 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:56.155748 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:56.201553 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:56.201513 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-b81a7a-78746cfd9c-ql62c" podUID="55a5e296-29c3-49eb-b79f-63378924400a" containerName="payload-processing" containerID="cri-o://87dbf0767f3c3368685fd177cf912f021d6c3abfd4fcfd31649baf4beef93103" gracePeriod=25 Oct 30 23:57:56.325147 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:56.325103 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b81a7a-5d9fb6cfb7-6wttp" podUID="aa45c557-0216-4475-9a34-e4a75bee03c3" containerName="payload-pre-processing" containerID="cri-o://52cb612c1ca2483cd95ca99a8a278ff41240c8ccda628e2f5980676172c94fd3" gracePeriod=25 Oct 30 23:57:56.714933 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:56.714905 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b81a7a-78746cfd9c-ql62c" Oct 30 23:57:56.748986 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:56.748964 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b81a7a-5d9fb6cfb7-6wttp" Oct 30 23:57:56.785876 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:56.785842 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55a5e296-29c3-49eb-b79f-63378924400a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55a5e296-29c3-49eb-b79f-63378924400a-plugins-config-volume\") pod \"55a5e296-29c3-49eb-b79f-63378924400a\" (UID: \"55a5e296-29c3-49eb-b79f-63378924400a\") " Oct 30 23:57:56.786051 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:56.785920 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55a5e296-29c3-49eb-b79f-63378924400a-kube-api-access-t8vdb\" (UniqueName: \"kubernetes.io/projected/55a5e296-29c3-49eb-b79f-63378924400a-kube-api-access-t8vdb\") pod \"55a5e296-29c3-49eb-b79f-63378924400a\" (UID: \"55a5e296-29c3-49eb-b79f-63378924400a\") " Oct 30 23:57:56.786134 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:56.786067 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa45c557-0216-4475-9a34-e4a75bee03c3-kube-api-access-92zgr\" (UniqueName: \"kubernetes.io/projected/aa45c557-0216-4475-9a34-e4a75bee03c3-kube-api-access-92zgr\") pod \"aa45c557-0216-4475-9a34-e4a75bee03c3\" (UID: \"aa45c557-0216-4475-9a34-e4a75bee03c3\") " Oct 30 23:57:56.786197 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:56.786158 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa45c557-0216-4475-9a34-e4a75bee03c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa45c557-0216-4475-9a34-e4a75bee03c3-plugins-config-volume\") pod \"aa45c557-0216-4475-9a34-e4a75bee03c3\" (UID: \"aa45c557-0216-4475-9a34-e4a75bee03c3\") " Oct 30 23:57:56.786333 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:56.786308 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55a5e296-29c3-49eb-b79f-63378924400a-plugins-config-volume" pod "55a5e296-29c3-49eb-b79f-63378924400a" (UID: "55a5e296-29c3-49eb-b79f-63378924400a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:57:56.786488 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:56.786466 4952 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55a5e296-29c3-49eb-b79f-63378924400a-plugins-config-volume\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:56.786759 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:56.786537 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa45c557-0216-4475-9a34-e4a75bee03c3-plugins-config-volume" pod "aa45c557-0216-4475-9a34-e4a75bee03c3" (UID: "aa45c557-0216-4475-9a34-e4a75bee03c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:57:56.788323 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:56.788299 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55a5e296-29c3-49eb-b79f-63378924400a-kube-api-access-t8vdb" pod "55a5e296-29c3-49eb-b79f-63378924400a" (UID: "55a5e296-29c3-49eb-b79f-63378924400a"). InnerVolumeSpecName "kube-api-access-t8vdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:57:56.788440 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:56.788421 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa45c557-0216-4475-9a34-e4a75bee03c3-kube-api-access-92zgr" pod "aa45c557-0216-4475-9a34-e4a75bee03c3" (UID: "aa45c557-0216-4475-9a34-e4a75bee03c3"). InnerVolumeSpecName "kube-api-access-92zgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:57:56.887956 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:56.887919 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8vdb\" (UniqueName: \"kubernetes.io/projected/55a5e296-29c3-49eb-b79f-63378924400a-kube-api-access-t8vdb\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:56.887956 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:56.887948 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92zgr\" (UniqueName: \"kubernetes.io/projected/aa45c557-0216-4475-9a34-e4a75bee03c3-kube-api-access-92zgr\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:56.887956 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:56.887962 4952 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa45c557-0216-4475-9a34-e4a75bee03c3-plugins-config-volume\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:57.121558 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:57.121526 4952 generic.go:356] "Generic (PLEG): container finished" podID="55a5e296-29c3-49eb-b79f-63378924400a" containerID="87dbf0767f3c3368685fd177cf912f021d6c3abfd4fcfd31649baf4beef93103" exitCode=0 Oct 30 23:57:57.121728 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:57.121612 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b81a7a-78746cfd9c-ql62c" Oct 30 23:57:57.121728 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:57.121624 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b81a7a-78746cfd9c-ql62c" event={"ID":"55a5e296-29c3-49eb-b79f-63378924400a","Type":"ContainerDied","Data":"87dbf0767f3c3368685fd177cf912f021d6c3abfd4fcfd31649baf4beef93103"} Oct 30 23:57:57.121728 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:57.121683 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b81a7a-78746cfd9c-ql62c" event={"ID":"55a5e296-29c3-49eb-b79f-63378924400a","Type":"ContainerDied","Data":"ace6a442078120874b6c467f0ebe09b3a81f11d8ef0b76dc5169fd5652e8d247"} Oct 30 23:57:57.121728 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:57.121706 4952 scope.go:122] "RemoveContainer" containerID="87dbf0767f3c3368685fd177cf912f021d6c3abfd4fcfd31649baf4beef93103" Oct 30 23:57:57.126853 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:57.123176 4952 generic.go:356] "Generic (PLEG): container finished" podID="aa45c557-0216-4475-9a34-e4a75bee03c3" containerID="52cb612c1ca2483cd95ca99a8a278ff41240c8ccda628e2f5980676172c94fd3" exitCode=0 Oct 30 23:57:57.126853 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:57.123217 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b81a7a-5d9fb6cfb7-6wttp" event={"ID":"aa45c557-0216-4475-9a34-e4a75bee03c3","Type":"ContainerDied","Data":"52cb612c1ca2483cd95ca99a8a278ff41240c8ccda628e2f5980676172c94fd3"} Oct 30 23:57:57.126853 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:57.123239 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b81a7a-5d9fb6cfb7-6wttp" Oct 30 23:57:57.126853 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:57.123259 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b81a7a-5d9fb6cfb7-6wttp" event={"ID":"aa45c557-0216-4475-9a34-e4a75bee03c3","Type":"ContainerDied","Data":"11522963f7763ae268f22f833693074cf534966bbbda646d2249056e46a42ecf"} Oct 30 23:57:57.134532 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:57.134515 4952 scope.go:122] "RemoveContainer" containerID="87dbf0767f3c3368685fd177cf912f021d6c3abfd4fcfd31649baf4beef93103" Oct 30 23:57:57.134885 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:57:57.134854 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87dbf0767f3c3368685fd177cf912f021d6c3abfd4fcfd31649baf4beef93103\": container with ID starting with 87dbf0767f3c3368685fd177cf912f021d6c3abfd4fcfd31649baf4beef93103 not found: ID does not exist" containerID="87dbf0767f3c3368685fd177cf912f021d6c3abfd4fcfd31649baf4beef93103" Oct 30 23:57:57.134945 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:57.134903 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87dbf0767f3c3368685fd177cf912f021d6c3abfd4fcfd31649baf4beef93103"} err="failed to get container status \"87dbf0767f3c3368685fd177cf912f021d6c3abfd4fcfd31649baf4beef93103\": rpc error: code = NotFound desc = could not find container \"87dbf0767f3c3368685fd177cf912f021d6c3abfd4fcfd31649baf4beef93103\": container with ID starting with 87dbf0767f3c3368685fd177cf912f021d6c3abfd4fcfd31649baf4beef93103 not found: ID does not exist" Oct 30 23:57:57.134945 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:57.134926 4952 scope.go:122] "RemoveContainer" containerID="52cb612c1ca2483cd95ca99a8a278ff41240c8ccda628e2f5980676172c94fd3" Oct 30 23:57:57.144245 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:57.144228 4952 scope.go:122] "RemoveContainer" containerID="52cb612c1ca2483cd95ca99a8a278ff41240c8ccda628e2f5980676172c94fd3" Oct 30 23:57:57.144518 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:57:57.144492 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52cb612c1ca2483cd95ca99a8a278ff41240c8ccda628e2f5980676172c94fd3\": container with ID starting with 52cb612c1ca2483cd95ca99a8a278ff41240c8ccda628e2f5980676172c94fd3 not found: ID does not exist" containerID="52cb612c1ca2483cd95ca99a8a278ff41240c8ccda628e2f5980676172c94fd3" Oct 30 23:57:57.144567 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:57.144529 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52cb612c1ca2483cd95ca99a8a278ff41240c8ccda628e2f5980676172c94fd3"} err="failed to get container status \"52cb612c1ca2483cd95ca99a8a278ff41240c8ccda628e2f5980676172c94fd3\": rpc error: code = NotFound desc = could not find container \"52cb612c1ca2483cd95ca99a8a278ff41240c8ccda628e2f5980676172c94fd3\": container with ID starting with 52cb612c1ca2483cd95ca99a8a278ff41240c8ccda628e2f5980676172c94fd3 not found: ID does not exist" Oct 30 23:57:57.289559 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:57.289480 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b81a7a-78746cfd9c-ql62c"] Oct 30 23:57:57.296413 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:57.296387 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b81a7a-78746cfd9c-ql62c"] Oct 30 23:57:57.405700 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:57.405665 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b81a7a-5d9fb6cfb7-6wttp"] Oct 30 23:57:57.410306 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:57.410277 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b81a7a-5d9fb6cfb7-6wttp"] Oct 30 23:57:57.973946 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:57.973910 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:58.222161 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:58.222125 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:57:58.968941 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:58.968911 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55a5e296-29c3-49eb-b79f-63378924400a" path="/var/lib/kubelet/pods/55a5e296-29c3-49eb-b79f-63378924400a/volumes" Oct 30 23:57:58.969356 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:58.969291 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa45c557-0216-4475-9a34-e4a75bee03c3" path="/var/lib/kubelet/pods/aa45c557-0216-4475-9a34-e4a75bee03c3/volumes" Oct 30 23:57:59.151941 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.151903 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" podUID="ab5286a5-6020-4c0b-9511-db63d4472a0a" containerName="main" containerID="cri-o://6072ad05f62dd188fd1a506af9929ba3000f2bc3cd3bac0fe3c40e33cbd73c84" gracePeriod=45 Oct 30 23:57:59.200571 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.200539 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" podUID="e662ea1f-f4ae-43ff-852c-ed3d6d282c74" containerName="main" containerID="cri-o://95c3dfecf98b9676e101eb7ffbebb99285ccb2779309883a2f70c501b0052016" gracePeriod=45 Oct 30 23:57:59.404494 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.404464 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8608b102-8sl2j"] Oct 30 23:57:59.404937 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.404919 4952 cpu_manager.go:414] "containerMap: removing container" podUID="145e7c01-efec-46e6-a748-56b39de76f83" containerName="main" Oct 30 23:57:59.405055 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.404940 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="145e7c01-efec-46e6-a748-56b39de76f83" containerName="main" Oct 30 23:57:59.405055 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.404957 4952 cpu_manager.go:414] "containerMap: removing container" podUID="789348bc-4698-479b-b2b0-da808059305f" containerName="revoke-keys" Oct 30 23:57:59.405055 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.404964 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="789348bc-4698-479b-b2b0-da808059305f" containerName="revoke-keys" Oct 30 23:57:59.405055 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.404980 4952 cpu_manager.go:414] "containerMap: removing container" podUID="aa45c557-0216-4475-9a34-e4a75bee03c3" containerName="payload-pre-processing" Oct 30 23:57:59.405055 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.404988 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa45c557-0216-4475-9a34-e4a75bee03c3" containerName="payload-pre-processing" Oct 30 23:57:59.405055 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.405004 4952 cpu_manager.go:414] "containerMap: removing container" podUID="55a5e296-29c3-49eb-b79f-63378924400a" containerName="payload-processing" Oct 30 23:57:59.405055 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.405011 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55a5e296-29c3-49eb-b79f-63378924400a" containerName="payload-processing" Oct 30 23:57:59.405409 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.405071 4952 cpu_manager.go:414] "containerMap: removing container" podUID="9fa21217-0585-4176-8a68-b7ec85bf1068" containerName="payload-pre-processing" Oct 30 23:57:59.405409 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.405080 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fa21217-0585-4176-8a68-b7ec85bf1068" containerName="payload-pre-processing" Oct 30 23:57:59.405409 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.405106 4952 cpu_manager.go:414] "containerMap: removing container" podUID="959b92af-9035-4dbd-aed3-5db4daaa63a8" containerName="main" Oct 30 23:57:59.405409 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.405113 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="959b92af-9035-4dbd-aed3-5db4daaa63a8" containerName="main" Oct 30 23:57:59.405409 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.405125 4952 cpu_manager.go:414] "containerMap: removing container" podUID="314cf6d3-fdc9-4496-b4de-2e531820361c" containerName="revoke-keys" Oct 30 23:57:59.405409 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.405133 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="314cf6d3-fdc9-4496-b4de-2e531820361c" containerName="revoke-keys" Oct 30 23:57:59.405409 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.405242 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="959b92af-9035-4dbd-aed3-5db4daaa63a8" containerName="main" Oct 30 23:57:59.405409 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.405255 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="314cf6d3-fdc9-4496-b4de-2e531820361c" containerName="revoke-keys" Oct 30 23:57:59.405409 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.405265 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="145e7c01-efec-46e6-a748-56b39de76f83" containerName="main" Oct 30 23:57:59.405409 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.405273 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="789348bc-4698-479b-b2b0-da808059305f" containerName="revoke-keys" Oct 30 23:57:59.405409 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.405283 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa45c557-0216-4475-9a34-e4a75bee03c3" containerName="payload-pre-processing" Oct 30 23:57:59.405409 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.405294 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="55a5e296-29c3-49eb-b79f-63378924400a" containerName="payload-processing" Oct 30 23:57:59.405409 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.405310 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fa21217-0585-4176-8a68-b7ec85bf1068" containerName="payload-pre-processing" Oct 30 23:57:59.408719 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.408697 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8608b102-8sl2j" Oct 30 23:57:59.462429 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.462404 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 23:57:59.463103 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.463080 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" Oct 30 23:57:59.463200 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.463171 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\"" Oct 30 23:57:59.484933 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.484886 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" Oct 30 23:57:59.514643 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.514611 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5286a5-6020-4c0b-9511-db63d4472a0a-home\" (UniqueName: \"kubernetes.io/empty-dir/ab5286a5-6020-4c0b-9511-db63d4472a0a-home\") pod \"ab5286a5-6020-4c0b-9511-db63d4472a0a\" (UID: \"ab5286a5-6020-4c0b-9511-db63d4472a0a\") " Oct 30 23:57:59.514784 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.514674 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-dshm\") pod \"e662ea1f-f4ae-43ff-852c-ed3d6d282c74\" (UID: \"e662ea1f-f4ae-43ff-852c-ed3d6d282c74\") " Oct 30 23:57:59.514784 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.514713 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab5286a5-6020-4c0b-9511-db63d4472a0a-kube-api-access-jvsk2\" (UniqueName: \"kubernetes.io/projected/ab5286a5-6020-4c0b-9511-db63d4472a0a-kube-api-access-jvsk2\") pod \"ab5286a5-6020-4c0b-9511-db63d4472a0a\" (UID: \"ab5286a5-6020-4c0b-9511-db63d4472a0a\") " Oct 30 23:57:59.514784 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.514744 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5286a5-6020-4c0b-9511-db63d4472a0a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab5286a5-6020-4c0b-9511-db63d4472a0a-dshm\") pod \"ab5286a5-6020-4c0b-9511-db63d4472a0a\" (UID: \"ab5286a5-6020-4c0b-9511-db63d4472a0a\") " Oct 30 23:57:59.514944 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.514805 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-kube-api-access-85cqg\" (UniqueName: \"kubernetes.io/projected/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-kube-api-access-85cqg\") pod \"e662ea1f-f4ae-43ff-852c-ed3d6d282c74\" (UID: \"e662ea1f-f4ae-43ff-852c-ed3d6d282c74\") " Oct 30 23:57:59.514944 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.514856 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab5286a5-6020-4c0b-9511-db63d4472a0a-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab5286a5-6020-4c0b-9511-db63d4472a0a-tls-certs\") pod \"ab5286a5-6020-4c0b-9511-db63d4472a0a\" (UID: \"ab5286a5-6020-4c0b-9511-db63d4472a0a\") " Oct 30 23:57:59.514944 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.514892 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-tmp-dir\") pod \"e662ea1f-f4ae-43ff-852c-ed3d6d282c74\" (UID: \"e662ea1f-f4ae-43ff-852c-ed3d6d282c74\") " Oct 30 23:57:59.514944 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.514918 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-home\" (UniqueName: \"kubernetes.io/empty-dir/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-home\") pod \"e662ea1f-f4ae-43ff-852c-ed3d6d282c74\" (UID: \"e662ea1f-f4ae-43ff-852c-ed3d6d282c74\") " Oct 30 23:57:59.515176 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.514946 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-tls-certs\" (UniqueName: \"kubernetes.io/secret/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-tls-certs\") pod \"e662ea1f-f4ae-43ff-852c-ed3d6d282c74\" (UID: \"e662ea1f-f4ae-43ff-852c-ed3d6d282c74\") " Oct 30 23:57:59.515176 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.514979 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5286a5-6020-4c0b-9511-db63d4472a0a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab5286a5-6020-4c0b-9511-db63d4472a0a-model-cache\") pod \"ab5286a5-6020-4c0b-9511-db63d4472a0a\" (UID: \"ab5286a5-6020-4c0b-9511-db63d4472a0a\") " Oct 30 23:57:59.515176 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.515073 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-model-cache\") pod \"e662ea1f-f4ae-43ff-852c-ed3d6d282c74\" (UID: \"e662ea1f-f4ae-43ff-852c-ed3d6d282c74\") " Oct 30 23:57:59.515176 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.515145 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5286a5-6020-4c0b-9511-db63d4472a0a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab5286a5-6020-4c0b-9511-db63d4472a0a-tmp-dir\") pod \"ab5286a5-6020-4c0b-9511-db63d4472a0a\" (UID: \"ab5286a5-6020-4c0b-9511-db63d4472a0a\") " Oct 30 23:57:59.515382 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.515309 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28642158-9697-4b4a-b1f2-55132b8eeeaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8608b102-8sl2j\" (UID: \"28642158-9697-4b4a-b1f2-55132b8eeeaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8608b102-8sl2j" Oct 30 23:57:59.515439 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.515405 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28642158-9697-4b4a-b1f2-55132b8eeeaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8608b102-8sl2j\" (UID: \"28642158-9697-4b4a-b1f2-55132b8eeeaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8608b102-8sl2j" Oct 30 23:57:59.515491 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.515447 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s84xd\" (UniqueName: \"kubernetes.io/projected/28642158-9697-4b4a-b1f2-55132b8eeeaf-kube-api-access-s84xd\") pod \"maas-api-revoke-sub-8608b102-8sl2j\" (UID: \"28642158-9697-4b4a-b1f2-55132b8eeeaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8608b102-8sl2j" Oct 30 23:57:59.515817 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.515792 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5286a5-6020-4c0b-9511-db63d4472a0a-home" pod "ab5286a5-6020-4c0b-9511-db63d4472a0a" (UID: "ab5286a5-6020-4c0b-9511-db63d4472a0a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:59.517341 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.517310 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5286a5-6020-4c0b-9511-db63d4472a0a-tmp-dir" pod "ab5286a5-6020-4c0b-9511-db63d4472a0a" (UID: "ab5286a5-6020-4c0b-9511-db63d4472a0a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:59.517548 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.517525 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5286a5-6020-4c0b-9511-db63d4472a0a-model-cache" pod "ab5286a5-6020-4c0b-9511-db63d4472a0a" (UID: "ab5286a5-6020-4c0b-9511-db63d4472a0a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:59.517628 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.517590 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-model-cache" pod "e662ea1f-f4ae-43ff-852c-ed3d6d282c74" (UID: "e662ea1f-f4ae-43ff-852c-ed3d6d282c74"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:59.517694 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.517636 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-home" pod "e662ea1f-f4ae-43ff-852c-ed3d6d282c74" (UID: "e662ea1f-f4ae-43ff-852c-ed3d6d282c74"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:59.517782 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.517759 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8608b102-8sl2j"] Oct 30 23:57:59.517866 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.517842 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-tmp-dir" pod "e662ea1f-f4ae-43ff-852c-ed3d6d282c74" (UID: "e662ea1f-f4ae-43ff-852c-ed3d6d282c74"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:59.520842 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.520807 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab5286a5-6020-4c0b-9511-db63d4472a0a-tls-certs" pod "ab5286a5-6020-4c0b-9511-db63d4472a0a" (UID: "ab5286a5-6020-4c0b-9511-db63d4472a0a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:57:59.520948 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.520857 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5286a5-6020-4c0b-9511-db63d4472a0a-dshm" pod "ab5286a5-6020-4c0b-9511-db63d4472a0a" (UID: "ab5286a5-6020-4c0b-9511-db63d4472a0a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:59.521215 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.521189 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-kube-api-access-85cqg" pod "e662ea1f-f4ae-43ff-852c-ed3d6d282c74" (UID: "e662ea1f-f4ae-43ff-852c-ed3d6d282c74"). InnerVolumeSpecName "kube-api-access-85cqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:57:59.524753 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.523794 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-dshm" pod "e662ea1f-f4ae-43ff-852c-ed3d6d282c74" (UID: "e662ea1f-f4ae-43ff-852c-ed3d6d282c74"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:59.525584 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.525555 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-tls-certs" pod "e662ea1f-f4ae-43ff-852c-ed3d6d282c74" (UID: "e662ea1f-f4ae-43ff-852c-ed3d6d282c74"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:57:59.526835 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.526813 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5286a5-6020-4c0b-9511-db63d4472a0a-kube-api-access-jvsk2" pod "ab5286a5-6020-4c0b-9511-db63d4472a0a" (UID: "ab5286a5-6020-4c0b-9511-db63d4472a0a"). InnerVolumeSpecName "kube-api-access-jvsk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:57:59.615965 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.615933 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28642158-9697-4b4a-b1f2-55132b8eeeaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8608b102-8sl2j\" (UID: \"28642158-9697-4b4a-b1f2-55132b8eeeaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8608b102-8sl2j" Oct 30 23:57:59.616136 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.615973 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s84xd\" (UniqueName: \"kubernetes.io/projected/28642158-9697-4b4a-b1f2-55132b8eeeaf-kube-api-access-s84xd\") pod \"maas-api-revoke-sub-8608b102-8sl2j\" (UID: \"28642158-9697-4b4a-b1f2-55132b8eeeaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8608b102-8sl2j" Oct 30 23:57:59.616136 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.616046 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28642158-9697-4b4a-b1f2-55132b8eeeaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8608b102-8sl2j\" (UID: \"28642158-9697-4b4a-b1f2-55132b8eeeaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8608b102-8sl2j" Oct 30 23:57:59.616136 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.616118 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvsk2\" (UniqueName: \"kubernetes.io/projected/ab5286a5-6020-4c0b-9511-db63d4472a0a-kube-api-access-jvsk2\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:59.616136 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.616132 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab5286a5-6020-4c0b-9511-db63d4472a0a-dshm\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:59.616272 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.616147 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85cqg\" (UniqueName: \"kubernetes.io/projected/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-kube-api-access-85cqg\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:59.616272 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.616161 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab5286a5-6020-4c0b-9511-db63d4472a0a-tls-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:59.616272 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.616175 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-tmp-dir\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:59.616272 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.616188 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-home\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:59.616272 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.616200 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-tls-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:59.616272 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.616212 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab5286a5-6020-4c0b-9511-db63d4472a0a-model-cache\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:59.616272 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.616226 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-model-cache\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:59.616272 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.616239 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab5286a5-6020-4c0b-9511-db63d4472a0a-tmp-dir\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:59.616272 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.616251 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab5286a5-6020-4c0b-9511-db63d4472a0a-home\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:59.616272 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.616263 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e662ea1f-f4ae-43ff-852c-ed3d6d282c74-dshm\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:57:59.616733 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.616712 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28642158-9697-4b4a-b1f2-55132b8eeeaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8608b102-8sl2j\" (UID: \"28642158-9697-4b4a-b1f2-55132b8eeeaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8608b102-8sl2j" Oct 30 23:57:59.618422 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.618390 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28642158-9697-4b4a-b1f2-55132b8eeeaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8608b102-8sl2j\" (UID: \"28642158-9697-4b4a-b1f2-55132b8eeeaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8608b102-8sl2j" Oct 30 23:57:59.674640 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.674611 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s84xd\" (UniqueName: \"kubernetes.io/projected/28642158-9697-4b4a-b1f2-55132b8eeeaf-kube-api-access-s84xd\") pod \"maas-api-revoke-sub-8608b102-8sl2j\" (UID: \"28642158-9697-4b4a-b1f2-55132b8eeeaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8608b102-8sl2j" Oct 30 23:57:59.725374 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:57:59.725349 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8608b102-8sl2j" Oct 30 23:58:00.110328 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:00.110306 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8608b102-8sl2j"] Oct 30 23:58:00.111317 ip-10-0-130-177 kubenswrapper[4952]: W1030 23:58:00.111292 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28642158_9697_4b4a_b1f2_55132b8eeeaf.slice/crio-462bad09353df3705f1d710a721c5357e979a410139cfa4db602a11968f75b2c WatchSource:0}: Error finding container 462bad09353df3705f1d710a721c5357e979a410139cfa4db602a11968f75b2c: Status 404 returned error can't find the container with id 462bad09353df3705f1d710a721c5357e979a410139cfa4db602a11968f75b2c Oct 30 23:58:00.115375 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:00.115355 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:00.144099 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:00.144077 4952 generic.go:356] "Generic (PLEG): container finished" podID="e662ea1f-f4ae-43ff-852c-ed3d6d282c74" containerID="95c3dfecf98b9676e101eb7ffbebb99285ccb2779309883a2f70c501b0052016" exitCode=0 Oct 30 23:58:00.144179 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:00.144117 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" Oct 30 23:58:00.144179 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:00.144156 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" event={"ID":"e662ea1f-f4ae-43ff-852c-ed3d6d282c74","Type":"ContainerDied","Data":"95c3dfecf98b9676e101eb7ffbebb99285ccb2779309883a2f70c501b0052016"} Oct 30 23:58:00.144299 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:00.144193 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb" event={"ID":"e662ea1f-f4ae-43ff-852c-ed3d6d282c74","Type":"ContainerDied","Data":"a9ff920794254c4706e9b411a6f8b5144517f325bec83a9d2dd1e4be75339bc4"} Oct 30 23:58:00.144299 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:00.144211 4952 scope.go:122] "RemoveContainer" containerID="95c3dfecf98b9676e101eb7ffbebb99285ccb2779309883a2f70c501b0052016" Oct 30 23:58:00.145550 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:00.145524 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8608b102-8sl2j" event={"ID":"28642158-9697-4b4a-b1f2-55132b8eeeaf","Type":"ContainerStarted","Data":"462bad09353df3705f1d710a721c5357e979a410139cfa4db602a11968f75b2c"} Oct 30 23:58:00.147211 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:00.147184 4952 generic.go:356] "Generic (PLEG): container finished" podID="ab5286a5-6020-4c0b-9511-db63d4472a0a" containerID="6072ad05f62dd188fd1a506af9929ba3000f2bc3cd3bac0fe3c40e33cbd73c84" exitCode=0 Oct 30 23:58:00.147339 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:00.147223 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" event={"ID":"ab5286a5-6020-4c0b-9511-db63d4472a0a","Type":"ContainerDied","Data":"6072ad05f62dd188fd1a506af9929ba3000f2bc3cd3bac0fe3c40e33cbd73c84"} Oct 30 23:58:00.147339 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:00.147231 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" Oct 30 23:58:00.147339 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:00.147250 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj" event={"ID":"ab5286a5-6020-4c0b-9511-db63d4472a0a","Type":"ContainerDied","Data":"a4c6070e1cf580aa66ea2590181191ced825c2ef3ebf33048a2bb5641576fd5c"} Oct 30 23:58:00.159725 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:00.159622 4952 scope.go:122] "RemoveContainer" containerID="95c3dfecf98b9676e101eb7ffbebb99285ccb2779309883a2f70c501b0052016" Oct 30 23:58:00.160289 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:58:00.160254 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95c3dfecf98b9676e101eb7ffbebb99285ccb2779309883a2f70c501b0052016\": container with ID starting with 95c3dfecf98b9676e101eb7ffbebb99285ccb2779309883a2f70c501b0052016 not found: ID does not exist" containerID="95c3dfecf98b9676e101eb7ffbebb99285ccb2779309883a2f70c501b0052016" Oct 30 23:58:00.160388 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:00.160305 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95c3dfecf98b9676e101eb7ffbebb99285ccb2779309883a2f70c501b0052016"} err="failed to get container status \"95c3dfecf98b9676e101eb7ffbebb99285ccb2779309883a2f70c501b0052016\": rpc error: code = NotFound desc = could not find container \"95c3dfecf98b9676e101eb7ffbebb99285ccb2779309883a2f70c501b0052016\": container with ID starting with 95c3dfecf98b9676e101eb7ffbebb99285ccb2779309883a2f70c501b0052016 not found: ID does not exist" Oct 30 23:58:00.160388 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:00.160330 4952 scope.go:122] "RemoveContainer" containerID="6072ad05f62dd188fd1a506af9929ba3000f2bc3cd3bac0fe3c40e33cbd73c84" Oct 30 23:58:00.183281 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:00.183257 4952 scope.go:122] "RemoveContainer" containerID="6072ad05f62dd188fd1a506af9929ba3000f2bc3cd3bac0fe3c40e33cbd73c84" Oct 30 23:58:00.183592 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:58:00.183571 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6072ad05f62dd188fd1a506af9929ba3000f2bc3cd3bac0fe3c40e33cbd73c84\": container with ID starting with 6072ad05f62dd188fd1a506af9929ba3000f2bc3cd3bac0fe3c40e33cbd73c84 not found: ID does not exist" containerID="6072ad05f62dd188fd1a506af9929ba3000f2bc3cd3bac0fe3c40e33cbd73c84" Oct 30 23:58:00.183667 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:00.183605 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6072ad05f62dd188fd1a506af9929ba3000f2bc3cd3bac0fe3c40e33cbd73c84"} err="failed to get container status \"6072ad05f62dd188fd1a506af9929ba3000f2bc3cd3bac0fe3c40e33cbd73c84\": rpc error: code = NotFound desc = could not find container \"6072ad05f62dd188fd1a506af9929ba3000f2bc3cd3bac0fe3c40e33cbd73c84\": container with ID starting with 6072ad05f62dd188fd1a506af9929ba3000f2bc3cd3bac0fe3c40e33cbd73c84 not found: ID does not exist" Oct 30 23:58:00.311200 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:00.311167 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb"] Oct 30 23:58:00.317126 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:00.317099 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b81a7a/simulator-w1-b81a7a-kserve-8f7f4769b-hdgcb"] Oct 30 23:58:00.425866 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:00.425846 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj"] Oct 30 23:58:00.432391 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:00.432370 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b81a7a/distinct-w1-b81a7a-kserve-b669bb8f6-bdqsj"] Oct 30 23:58:00.970883 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:00.970857 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab5286a5-6020-4c0b-9511-db63d4472a0a" path="/var/lib/kubelet/pods/ab5286a5-6020-4c0b-9511-db63d4472a0a/volumes" Oct 30 23:58:00.971264 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:00.971251 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e662ea1f-f4ae-43ff-852c-ed3d6d282c74" path="/var/lib/kubelet/pods/e662ea1f-f4ae-43ff-852c-ed3d6d282c74/volumes" Oct 30 23:58:01.155514 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:01.155482 4952 generic.go:356] "Generic (PLEG): container finished" podID="28642158-9697-4b4a-b1f2-55132b8eeeaf" containerID="7534751ae85c1cdba3154aaac2c54ef545fb71d5278d11ac065db4cdf4d3e3e1" exitCode=0 Oct 30 23:58:01.155971 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:01.155564 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8608b102-8sl2j" event={"ID":"28642158-9697-4b4a-b1f2-55132b8eeeaf","Type":"ContainerDied","Data":"7534751ae85c1cdba3154aaac2c54ef545fb71d5278d11ac065db4cdf4d3e3e1"} Oct 30 23:58:01.504744 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:01.504717 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:01.801312 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:01.801223 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:01.969081 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:01.969027 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh"] Oct 30 23:58:01.969261 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:01.969240 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" podUID="d4257e26-e76e-49cc-bbd5-27c7e8075a5e" containerName="istio-proxy" containerID="cri-o://1420dfd58e348bcddbeaaa851e634d90f0afff6d55206efce39eefad251555b7" gracePeriod=30 Oct 30 23:58:02.140126 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:02.140098 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:02.298663 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:02.298640 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8608b102-8sl2j" Oct 30 23:58:02.338283 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:02.338261 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28642158-9697-4b4a-b1f2-55132b8eeeaf-kube-api-access-s84xd\" (UniqueName: \"kubernetes.io/projected/28642158-9697-4b4a-b1f2-55132b8eeeaf-kube-api-access-s84xd\") pod \"28642158-9697-4b4a-b1f2-55132b8eeeaf\" (UID: \"28642158-9697-4b4a-b1f2-55132b8eeeaf\") " Oct 30 23:58:02.338395 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:02.338300 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28642158-9697-4b4a-b1f2-55132b8eeeaf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28642158-9697-4b4a-b1f2-55132b8eeeaf-maas-api-xks-ca\") pod \"28642158-9697-4b4a-b1f2-55132b8eeeaf\" (UID: \"28642158-9697-4b4a-b1f2-55132b8eeeaf\") " Oct 30 23:58:02.338395 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:02.338345 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28642158-9697-4b4a-b1f2-55132b8eeeaf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28642158-9697-4b4a-b1f2-55132b8eeeaf-maas-api-openshift-ca\") pod \"28642158-9697-4b4a-b1f2-55132b8eeeaf\" (UID: \"28642158-9697-4b4a-b1f2-55132b8eeeaf\") " Oct 30 23:58:02.338636 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:02.338616 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28642158-9697-4b4a-b1f2-55132b8eeeaf-maas-api-openshift-ca" pod "28642158-9697-4b4a-b1f2-55132b8eeeaf" (UID: "28642158-9697-4b4a-b1f2-55132b8eeeaf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:58:02.340501 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:02.340474 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28642158-9697-4b4a-b1f2-55132b8eeeaf-kube-api-access-s84xd" pod "28642158-9697-4b4a-b1f2-55132b8eeeaf" (UID: "28642158-9697-4b4a-b1f2-55132b8eeeaf"). InnerVolumeSpecName "kube-api-access-s84xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:02.340590 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:02.340543 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28642158-9697-4b4a-b1f2-55132b8eeeaf-maas-api-xks-ca" pod "28642158-9697-4b4a-b1f2-55132b8eeeaf" (UID: "28642158-9697-4b4a-b1f2-55132b8eeeaf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:58:02.439828 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:02.439763 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28642158-9697-4b4a-b1f2-55132b8eeeaf-maas-api-openshift-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:58:02.439828 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:02.439787 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s84xd\" (UniqueName: \"kubernetes.io/projected/28642158-9697-4b4a-b1f2-55132b8eeeaf-kube-api-access-s84xd\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:58:02.439828 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:02.439796 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28642158-9697-4b4a-b1f2-55132b8eeeaf-maas-api-xks-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:58:02.567151 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:02.567117 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:03.169001 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:03.168969 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8608b102-8sl2j" Oct 30 23:58:03.169187 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:03.169002 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8608b102-8sl2j" event={"ID":"28642158-9697-4b4a-b1f2-55132b8eeeaf","Type":"ContainerDied","Data":"462bad09353df3705f1d710a721c5357e979a410139cfa4db602a11968f75b2c"} Oct 30 23:58:03.169187 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:03.169081 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="462bad09353df3705f1d710a721c5357e979a410139cfa4db602a11968f75b2c" Oct 30 23:58:06.655017 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:06.654972 4952 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 23:58:06.655422 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:06.655067 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" podUID="d4257e26-e76e-49cc-bbd5-27c7e8075a5e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:58:07.194978 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:07.194945 4952 generic.go:356] "Generic (PLEG): container finished" podID="d4257e26-e76e-49cc-bbd5-27c7e8075a5e" containerID="1420dfd58e348bcddbeaaa851e634d90f0afff6d55206efce39eefad251555b7" exitCode=0 Oct 30 23:58:07.195183 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:07.195020 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" event={"ID":"d4257e26-e76e-49cc-bbd5-27c7e8075a5e","Type":"ContainerDied","Data":"1420dfd58e348bcddbeaaa851e634d90f0afff6d55206efce39eefad251555b7"} Oct 30 23:58:07.219552 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:07.219527 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" Oct 30 23:58:07.278209 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:07.278151 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-istio-token\" (UniqueName: \"kubernetes.io/projected/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-istio-token\") pod \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " Oct 30 23:58:07.278343 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:07.278217 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-istio-data\") pod \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " Oct 30 23:58:07.278343 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:07.278241 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-workload-socket\") pod \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " Oct 30 23:58:07.278343 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:07.278261 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-istiod-ca-cert\") pod \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " Oct 30 23:58:07.278343 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:07.278278 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-workload-certs\") pod \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " Oct 30 23:58:07.278343 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:07.278296 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-kube-api-access-5zrbk\" (UniqueName: \"kubernetes.io/projected/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-kube-api-access-5zrbk\") pod \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " Oct 30 23:58:07.278565 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:07.278355 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-credential-socket\") pod \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " Oct 30 23:58:07.278565 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:07.278388 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-istio-podinfo\") pod \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " Oct 30 23:58:07.278565 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:07.278416 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-istio-envoy\") pod \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\" (UID: \"d4257e26-e76e-49cc-bbd5-27c7e8075a5e\") " Oct 30 23:58:07.278565 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:07.278451 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-istio-data" pod "d4257e26-e76e-49cc-bbd5-27c7e8075a5e" (UID: "d4257e26-e76e-49cc-bbd5-27c7e8075a5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:07.278775 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:07.278540 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-workload-socket" pod "d4257e26-e76e-49cc-bbd5-27c7e8075a5e" (UID: "d4257e26-e76e-49cc-bbd5-27c7e8075a5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:07.278775 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:07.278624 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-credential-socket" pod "d4257e26-e76e-49cc-bbd5-27c7e8075a5e" (UID: "d4257e26-e76e-49cc-bbd5-27c7e8075a5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:07.278775 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:07.278641 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-workload-certs" pod "d4257e26-e76e-49cc-bbd5-27c7e8075a5e" (UID: "d4257e26-e76e-49cc-bbd5-27c7e8075a5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:07.278775 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:07.278758 4952 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-istio-data\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:58:07.278954 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:07.278780 4952 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-workload-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:58:07.278954 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:07.278797 4952 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-workload-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:58:07.278954 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:07.278811 4952 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-credential-socket\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:58:07.278954 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:07.278843 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-istiod-ca-cert" pod "d4257e26-e76e-49cc-bbd5-27c7e8075a5e" (UID: "d4257e26-e76e-49cc-bbd5-27c7e8075a5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:58:07.280765 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:07.280746 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-istio-envoy" pod "d4257e26-e76e-49cc-bbd5-27c7e8075a5e" (UID: "d4257e26-e76e-49cc-bbd5-27c7e8075a5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:58:07.280765 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:07.280750 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-istio-token" pod "d4257e26-e76e-49cc-bbd5-27c7e8075a5e" (UID: "d4257e26-e76e-49cc-bbd5-27c7e8075a5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:07.281169 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:07.281153 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-istio-podinfo" pod "d4257e26-e76e-49cc-bbd5-27c7e8075a5e" (UID: "d4257e26-e76e-49cc-bbd5-27c7e8075a5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:58:07.281169 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:07.281150 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-kube-api-access-5zrbk" pod "d4257e26-e76e-49cc-bbd5-27c7e8075a5e" (UID: "d4257e26-e76e-49cc-bbd5-27c7e8075a5e"). InnerVolumeSpecName "kube-api-access-5zrbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:07.379458 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:07.379433 4952 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-istio-podinfo\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:58:07.379458 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:07.379461 4952 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-istio-envoy\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:58:07.379633 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:07.379474 4952 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-istio-token\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:58:07.379633 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:07.379486 4952 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-istiod-ca-cert\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:58:07.379633 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:07.379500 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zrbk\" (UniqueName: \"kubernetes.io/projected/d4257e26-e76e-49cc-bbd5-27c7e8075a5e-kube-api-access-5zrbk\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:58:08.201685 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:08.201651 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" event={"ID":"d4257e26-e76e-49cc-bbd5-27c7e8075a5e","Type":"ContainerDied","Data":"d78165a3e0abe000363846ba9f6f290e37dc8c880d78fedf2b8170d66cc6fb0f"} Oct 30 23:58:08.201685 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:08.201686 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh" Oct 30 23:58:08.202196 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:08.201712 4952 scope.go:122] "RemoveContainer" containerID="1420dfd58e348bcddbeaaa851e634d90f0afff6d55206efce39eefad251555b7" Oct 30 23:58:08.416962 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:08.416933 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh"] Oct 30 23:58:08.466715 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:08.466657 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-zjwqh"] Oct 30 23:58:08.970012 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:08.969978 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4257e26-e76e-49cc-bbd5-27c7e8075a5e" path="/var/lib/kubelet/pods/d4257e26-e76e-49cc-bbd5-27c7e8075a5e/volumes" Oct 30 23:58:14.885004 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:14.884966 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:16.423571 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:16.423510 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:16.717377 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:16.717294 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:17.339118 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:17.339084 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:21.140963 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:58:21.140927 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:58:21.819817 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:21.819780 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:22.052992 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:22.052956 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:22.294030 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:22.293985 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:23.242364 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:23.242330 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:23.485690 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:23.485645 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:23.717475 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:23.717440 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:24.512577 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:24.512545 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:27.424231 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:27.424196 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 30 23:58:28.304771 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:28.304730 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:28.541724 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:28.541678 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:28.716669 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:28.716638 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7bbf6fc889-b89vm"] Oct 30 23:58:28.716852 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:28.716831 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7bbf6fc889-b89vm" podUID="da1faa77-ca82-428f-8d34-c602c4238182" containerName="manager" containerID="cri-o://1592739455dc16ebf7716590b945c88a14f75b5696d8045c88ee97d145b0cef7" gracePeriod=30 Oct 30 23:58:28.999906 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:28.999880 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbf6fc889-b89vm" Oct 30 23:58:29.071007 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:29.070979 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da1faa77-ca82-428f-8d34-c602c4238182-webhook-cert\" (UniqueName: \"kubernetes.io/secret/da1faa77-ca82-428f-8d34-c602c4238182-webhook-cert\") pod \"da1faa77-ca82-428f-8d34-c602c4238182\" (UID: \"da1faa77-ca82-428f-8d34-c602c4238182\") " Oct 30 23:58:29.071183 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:29.071030 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da1faa77-ca82-428f-8d34-c602c4238182-kube-api-access-wzcdp\" (UniqueName: \"kubernetes.io/projected/da1faa77-ca82-428f-8d34-c602c4238182-kube-api-access-wzcdp\") pod \"da1faa77-ca82-428f-8d34-c602c4238182\" (UID: \"da1faa77-ca82-428f-8d34-c602c4238182\") " Oct 30 23:58:29.071183 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:29.071159 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da1faa77-ca82-428f-8d34-c602c4238182-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da1faa77-ca82-428f-8d34-c602c4238182-metrics-certs\") pod \"da1faa77-ca82-428f-8d34-c602c4238182\" (UID: \"da1faa77-ca82-428f-8d34-c602c4238182\") " Oct 30 23:58:29.073345 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:29.073313 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da1faa77-ca82-428f-8d34-c602c4238182-webhook-cert" pod "da1faa77-ca82-428f-8d34-c602c4238182" (UID: "da1faa77-ca82-428f-8d34-c602c4238182"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:58:29.073445 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:29.073427 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da1faa77-ca82-428f-8d34-c602c4238182-metrics-certs" pod "da1faa77-ca82-428f-8d34-c602c4238182" (UID: "da1faa77-ca82-428f-8d34-c602c4238182"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:58:29.073529 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:29.073507 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da1faa77-ca82-428f-8d34-c602c4238182-kube-api-access-wzcdp" pod "da1faa77-ca82-428f-8d34-c602c4238182" (UID: "da1faa77-ca82-428f-8d34-c602c4238182"). InnerVolumeSpecName "kube-api-access-wzcdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:58:29.172952 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:29.172922 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzcdp\" (UniqueName: \"kubernetes.io/projected/da1faa77-ca82-428f-8d34-c602c4238182-kube-api-access-wzcdp\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:58:29.172952 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:29.172952 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da1faa77-ca82-428f-8d34-c602c4238182-metrics-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:58:29.173180 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:29.172962 4952 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/da1faa77-ca82-428f-8d34-c602c4238182-webhook-cert\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 23:58:29.388859 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:29.388819 4952 generic.go:356] "Generic (PLEG): container finished" podID="da1faa77-ca82-428f-8d34-c602c4238182" containerID="1592739455dc16ebf7716590b945c88a14f75b5696d8045c88ee97d145b0cef7" exitCode=0 Oct 30 23:58:29.389054 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:29.388865 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbf6fc889-b89vm" Oct 30 23:58:29.389054 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:29.388909 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbf6fc889-b89vm" event={"ID":"da1faa77-ca82-428f-8d34-c602c4238182","Type":"ContainerDied","Data":"1592739455dc16ebf7716590b945c88a14f75b5696d8045c88ee97d145b0cef7"} Oct 30 23:58:29.389054 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:29.388959 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbf6fc889-b89vm" event={"ID":"da1faa77-ca82-428f-8d34-c602c4238182","Type":"ContainerDied","Data":"061320028cce9f6c82a5fdfb3cd9ca71bb09cdd67c4442a53a410c484a60b434"} Oct 30 23:58:29.389054 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:29.388982 4952 scope.go:122] "RemoveContainer" containerID="1592739455dc16ebf7716590b945c88a14f75b5696d8045c88ee97d145b0cef7" Oct 30 23:58:29.399375 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:29.399358 4952 scope.go:122] "RemoveContainer" containerID="1592739455dc16ebf7716590b945c88a14f75b5696d8045c88ee97d145b0cef7" Oct 30 23:58:29.399644 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:58:29.399626 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1592739455dc16ebf7716590b945c88a14f75b5696d8045c88ee97d145b0cef7\": container with ID starting with 1592739455dc16ebf7716590b945c88a14f75b5696d8045c88ee97d145b0cef7 not found: ID does not exist" containerID="1592739455dc16ebf7716590b945c88a14f75b5696d8045c88ee97d145b0cef7" Oct 30 23:58:29.399689 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:29.399654 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1592739455dc16ebf7716590b945c88a14f75b5696d8045c88ee97d145b0cef7"} err="failed to get container status \"1592739455dc16ebf7716590b945c88a14f75b5696d8045c88ee97d145b0cef7\": rpc error: code = NotFound desc = could not find container \"1592739455dc16ebf7716590b945c88a14f75b5696d8045c88ee97d145b0cef7\": container with ID starting with 1592739455dc16ebf7716590b945c88a14f75b5696d8045c88ee97d145b0cef7 not found: ID does not exist" Oct 30 23:58:29.553196 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:29.553167 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7bbf6fc889-b89vm"] Oct 30 23:58:29.559367 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:29.559338 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7bbf6fc889-b89vm"] Oct 30 23:58:30.969502 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:30.969468 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da1faa77-ca82-428f-8d34-c602c4238182" path="/var/lib/kubelet/pods/da1faa77-ca82-428f-8d34-c602c4238182/volumes" Oct 30 23:58:37.297675 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:58:37.297634 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:51.141374 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:58:51.141338 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:59:06.694676 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:59:06.694626 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:08.841969 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:59:08.841937 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:21.141509 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:59:21.141463 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:59:28.660142 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:59:28.660102 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:51.141283 ip-10-0-130-177 kubenswrapper[4952]: E1030 23:59:51.141199 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 30 23:59:53.885737 ip-10-0-130-177 kubenswrapper[4952]: I1030 23:59:53.885701 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:00:02.388735 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:02.388700 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l8w2v"] Oct 31 00:00:02.389137 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:02.389076 4952 cpu_manager.go:414] "containerMap: removing container" podUID="28642158-9697-4b4a-b1f2-55132b8eeeaf" containerName="revoke-keys" Oct 31 00:00:02.389137 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:02.389087 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28642158-9697-4b4a-b1f2-55132b8eeeaf" containerName="revoke-keys" Oct 31 00:00:02.389137 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:02.389099 4952 cpu_manager.go:414] "containerMap: removing container" podUID="ab5286a5-6020-4c0b-9511-db63d4472a0a" containerName="main" Oct 31 00:00:02.389137 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:02.389104 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab5286a5-6020-4c0b-9511-db63d4472a0a" containerName="main" Oct 31 00:00:02.389137 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:02.389121 4952 cpu_manager.go:414] "containerMap: removing container" podUID="e662ea1f-f4ae-43ff-852c-ed3d6d282c74" containerName="main" Oct 31 00:00:02.389137 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:02.389126 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e662ea1f-f4ae-43ff-852c-ed3d6d282c74" containerName="main" Oct 31 00:00:02.389137 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:02.389134 4952 cpu_manager.go:414] "containerMap: removing container" podUID="d4257e26-e76e-49cc-bbd5-27c7e8075a5e" containerName="istio-proxy" Oct 31 00:00:02.389137 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:02.389139 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4257e26-e76e-49cc-bbd5-27c7e8075a5e" containerName="istio-proxy" Oct 31 00:00:02.389415 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:02.389174 4952 cpu_manager.go:414] "containerMap: removing container" podUID="da1faa77-ca82-428f-8d34-c602c4238182" containerName="manager" Oct 31 00:00:02.389415 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:02.389179 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da1faa77-ca82-428f-8d34-c602c4238182" containerName="manager" Oct 31 00:00:02.389415 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:02.389239 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="28642158-9697-4b4a-b1f2-55132b8eeeaf" containerName="revoke-keys" Oct 31 00:00:02.389415 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:02.389247 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab5286a5-6020-4c0b-9511-db63d4472a0a" containerName="main" Oct 31 00:00:02.389415 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:02.389253 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="e662ea1f-f4ae-43ff-852c-ed3d6d282c74" containerName="main" Oct 31 00:00:02.389415 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:02.389258 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4257e26-e76e-49cc-bbd5-27c7e8075a5e" containerName="istio-proxy" Oct 31 00:00:02.389415 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:02.389266 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="da1faa77-ca82-428f-8d34-c602c4238182" containerName="manager" Oct 31 00:00:02.392445 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:02.392429 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l8w2v" Oct 31 00:00:02.450183 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:02.450155 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l8w2v"] Oct 31 00:00:02.495865 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:02.495829 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt7fn\" (UniqueName: \"kubernetes.io/projected/a75e2636-1bbc-4dfd-a4a8-3a867026f118-kube-api-access-nt7fn\") pod \"maas-api-revoke-sub-c1e271a9-l8w2v\" (UID: \"a75e2636-1bbc-4dfd-a4a8-3a867026f118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l8w2v" Oct 31 00:00:02.495990 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:02.495885 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a75e2636-1bbc-4dfd-a4a8-3a867026f118-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-l8w2v\" (UID: \"a75e2636-1bbc-4dfd-a4a8-3a867026f118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l8w2v" Oct 31 00:00:02.496032 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:02.495979 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a75e2636-1bbc-4dfd-a4a8-3a867026f118-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-l8w2v\" (UID: \"a75e2636-1bbc-4dfd-a4a8-3a867026f118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l8w2v" Oct 31 00:00:02.496132 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:02.496114 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:00:02.496299 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:02.496285 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\"" Oct 31 00:00:02.597398 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:02.597372 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt7fn\" (UniqueName: \"kubernetes.io/projected/a75e2636-1bbc-4dfd-a4a8-3a867026f118-kube-api-access-nt7fn\") pod \"maas-api-revoke-sub-c1e271a9-l8w2v\" (UID: \"a75e2636-1bbc-4dfd-a4a8-3a867026f118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l8w2v" Oct 31 00:00:02.597561 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:02.597405 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a75e2636-1bbc-4dfd-a4a8-3a867026f118-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-l8w2v\" (UID: \"a75e2636-1bbc-4dfd-a4a8-3a867026f118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l8w2v" Oct 31 00:00:02.597561 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:02.597428 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a75e2636-1bbc-4dfd-a4a8-3a867026f118-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-l8w2v\" (UID: \"a75e2636-1bbc-4dfd-a4a8-3a867026f118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l8w2v" Oct 31 00:00:02.598193 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:02.598173 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a75e2636-1bbc-4dfd-a4a8-3a867026f118-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-l8w2v\" (UID: \"a75e2636-1bbc-4dfd-a4a8-3a867026f118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l8w2v" Oct 31 00:00:02.599955 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:02.599935 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a75e2636-1bbc-4dfd-a4a8-3a867026f118-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-l8w2v\" (UID: \"a75e2636-1bbc-4dfd-a4a8-3a867026f118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l8w2v" Oct 31 00:00:02.655218 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:02.655161 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt7fn\" (UniqueName: \"kubernetes.io/projected/a75e2636-1bbc-4dfd-a4a8-3a867026f118-kube-api-access-nt7fn\") pod \"maas-api-revoke-sub-c1e271a9-l8w2v\" (UID: \"a75e2636-1bbc-4dfd-a4a8-3a867026f118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l8w2v" Oct 31 00:00:02.723499 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:02.723467 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l8w2v" Oct 31 00:00:03.097026 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:03.096999 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l8w2v"] Oct 31 00:00:03.098906 ip-10-0-130-177 kubenswrapper[4952]: W1031 00:00:03.098876 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda75e2636_1bbc_4dfd_a4a8_3a867026f118.slice/crio-26377fa2d24bfa138636e689c91c06dc69729075c4e946d78f4c577b7cac71dc WatchSource:0}: Error finding container 26377fa2d24bfa138636e689c91c06dc69729075c4e946d78f4c577b7cac71dc: Status 404 returned error can't find the container with id 26377fa2d24bfa138636e689c91c06dc69729075c4e946d78f4c577b7cac71dc Oct 31 00:00:03.924180 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:03.924138 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l8w2v" event={"ID":"a75e2636-1bbc-4dfd-a4a8-3a867026f118","Type":"ContainerStarted","Data":"507ee0e6f988178e5662b6fdc5508f6dd69902f9204bce5a593ffad95c5e1133"} Oct 31 00:00:03.924546 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:03.924202 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l8w2v" event={"ID":"a75e2636-1bbc-4dfd-a4a8-3a867026f118","Type":"ContainerStarted","Data":"26377fa2d24bfa138636e689c91c06dc69729075c4e946d78f4c577b7cac71dc"} Oct 31 00:00:04.043850 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:04.043801 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l8w2v" podStartSLOduration=2.043790257 podStartE2EDuration="2.043790257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:04.042207914 +0000 UTC m=+3223.186038140" watchObservedRunningTime="2026-10-31 00:00:04.043790257 +0000 UTC m=+3223.187620501" Oct 31 00:00:04.492513 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:04.492484 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:00:04.931264 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:04.931235 4952 generic.go:356] "Generic (PLEG): container finished" podID="a75e2636-1bbc-4dfd-a4a8-3a867026f118" containerID="507ee0e6f988178e5662b6fdc5508f6dd69902f9204bce5a593ffad95c5e1133" exitCode=0 Oct 31 00:00:04.931611 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:04.931308 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l8w2v" event={"ID":"a75e2636-1bbc-4dfd-a4a8-3a867026f118","Type":"ContainerDied","Data":"507ee0e6f988178e5662b6fdc5508f6dd69902f9204bce5a593ffad95c5e1133"} Oct 31 00:00:06.072545 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:06.072520 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l8w2v" Oct 31 00:00:06.223734 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:06.223668 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a75e2636-1bbc-4dfd-a4a8-3a867026f118-kube-api-access-nt7fn\" (UniqueName: \"kubernetes.io/projected/a75e2636-1bbc-4dfd-a4a8-3a867026f118-kube-api-access-nt7fn\") pod \"a75e2636-1bbc-4dfd-a4a8-3a867026f118\" (UID: \"a75e2636-1bbc-4dfd-a4a8-3a867026f118\") " Oct 31 00:00:06.223875 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:06.223741 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a75e2636-1bbc-4dfd-a4a8-3a867026f118-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a75e2636-1bbc-4dfd-a4a8-3a867026f118-maas-api-xks-ca\") pod \"a75e2636-1bbc-4dfd-a4a8-3a867026f118\" (UID: \"a75e2636-1bbc-4dfd-a4a8-3a867026f118\") " Oct 31 00:00:06.223875 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:06.223781 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a75e2636-1bbc-4dfd-a4a8-3a867026f118-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a75e2636-1bbc-4dfd-a4a8-3a867026f118-maas-api-openshift-ca\") pod \"a75e2636-1bbc-4dfd-a4a8-3a867026f118\" (UID: \"a75e2636-1bbc-4dfd-a4a8-3a867026f118\") " Oct 31 00:00:06.224189 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:06.224160 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a75e2636-1bbc-4dfd-a4a8-3a867026f118-maas-api-openshift-ca" pod "a75e2636-1bbc-4dfd-a4a8-3a867026f118" (UID: "a75e2636-1bbc-4dfd-a4a8-3a867026f118"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:00:06.225899 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:06.225877 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a75e2636-1bbc-4dfd-a4a8-3a867026f118-maas-api-xks-ca" pod "a75e2636-1bbc-4dfd-a4a8-3a867026f118" (UID: "a75e2636-1bbc-4dfd-a4a8-3a867026f118"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:00:06.226164 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:06.226145 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a75e2636-1bbc-4dfd-a4a8-3a867026f118-kube-api-access-nt7fn" pod "a75e2636-1bbc-4dfd-a4a8-3a867026f118" (UID: "a75e2636-1bbc-4dfd-a4a8-3a867026f118"). InnerVolumeSpecName "kube-api-access-nt7fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:00:06.324990 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:06.324962 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a75e2636-1bbc-4dfd-a4a8-3a867026f118-maas-api-openshift-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:00:06.324990 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:06.324986 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt7fn\" (UniqueName: \"kubernetes.io/projected/a75e2636-1bbc-4dfd-a4a8-3a867026f118-kube-api-access-nt7fn\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:00:06.325186 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:06.324996 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a75e2636-1bbc-4dfd-a4a8-3a867026f118-maas-api-xks-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:00:06.943131 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:06.943101 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l8w2v" Oct 31 00:00:06.943131 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:06.943110 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l8w2v" event={"ID":"a75e2636-1bbc-4dfd-a4a8-3a867026f118","Type":"ContainerDied","Data":"26377fa2d24bfa138636e689c91c06dc69729075c4e946d78f4c577b7cac71dc"} Oct 31 00:00:06.943371 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:06.943186 4952 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26377fa2d24bfa138636e689c91c06dc69729075c4e946d78f4c577b7cac71dc" Oct 31 00:00:09.727761 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:09.727728 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn"] Oct 31 00:00:09.728158 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:09.728122 4952 cpu_manager.go:414] "containerMap: removing container" podUID="a75e2636-1bbc-4dfd-a4a8-3a867026f118" containerName="revoke-keys" Oct 31 00:00:09.728158 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:09.728133 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a75e2636-1bbc-4dfd-a4a8-3a867026f118" containerName="revoke-keys" Oct 31 00:00:09.728241 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:09.728202 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="a75e2636-1bbc-4dfd-a4a8-3a867026f118" containerName="revoke-keys" Oct 31 00:00:09.731384 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:09.731367 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" Oct 31 00:00:09.827649 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:09.827622 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn"] Oct 31 00:00:09.851160 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:09.851137 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2831d3-e0d4-4a4b-a209-d5cb09632279-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-v4ktn\" (UID: \"fd2831d3-e0d4-4a4b-a209-d5cb09632279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" Oct 31 00:00:09.851298 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:09.851166 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2831d3-e0d4-4a4b-a209-d5cb09632279-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-v4ktn\" (UID: \"fd2831d3-e0d4-4a4b-a209-d5cb09632279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" Oct 31 00:00:09.851298 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:09.851207 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2831d3-e0d4-4a4b-a209-d5cb09632279-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-v4ktn\" (UID: \"fd2831d3-e0d4-4a4b-a209-d5cb09632279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" Oct 31 00:00:09.851382 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:09.851308 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2831d3-e0d4-4a4b-a209-d5cb09632279-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-v4ktn\" (UID: \"fd2831d3-e0d4-4a4b-a209-d5cb09632279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" Oct 31 00:00:09.851425 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:09.851379 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd2831d3-e0d4-4a4b-a209-d5cb09632279-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-v4ktn\" (UID: \"fd2831d3-e0d4-4a4b-a209-d5cb09632279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" Oct 31 00:00:09.851425 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:09.851405 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2831d3-e0d4-4a4b-a209-d5cb09632279-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-v4ktn\" (UID: \"fd2831d3-e0d4-4a4b-a209-d5cb09632279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" Oct 31 00:00:09.851504 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:09.851466 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f28bj\" (UniqueName: \"kubernetes.io/projected/fd2831d3-e0d4-4a4b-a209-d5cb09632279-kube-api-access-f28bj\") pod \"maas-default-gateway-openshift-default-5df48fcc99-v4ktn\" (UID: \"fd2831d3-e0d4-4a4b-a209-d5cb09632279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" Oct 31 00:00:09.851504 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:09.851501 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2831d3-e0d4-4a4b-a209-d5cb09632279-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-v4ktn\" (UID: \"fd2831d3-e0d4-4a4b-a209-d5cb09632279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" Oct 31 00:00:09.851576 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:09.851518 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2831d3-e0d4-4a4b-a209-d5cb09632279-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-v4ktn\" (UID: \"fd2831d3-e0d4-4a4b-a209-d5cb09632279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" Oct 31 00:00:09.951796 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:09.951768 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2831d3-e0d4-4a4b-a209-d5cb09632279-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-v4ktn\" (UID: \"fd2831d3-e0d4-4a4b-a209-d5cb09632279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" Oct 31 00:00:09.951913 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:09.951799 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2831d3-e0d4-4a4b-a209-d5cb09632279-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-v4ktn\" (UID: \"fd2831d3-e0d4-4a4b-a209-d5cb09632279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" Oct 31 00:00:09.951913 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:09.951818 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2831d3-e0d4-4a4b-a209-d5cb09632279-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-v4ktn\" (UID: \"fd2831d3-e0d4-4a4b-a209-d5cb09632279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" Oct 31 00:00:09.951913 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:09.951832 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2831d3-e0d4-4a4b-a209-d5cb09632279-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-v4ktn\" (UID: \"fd2831d3-e0d4-4a4b-a209-d5cb09632279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" Oct 31 00:00:09.951913 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:09.951872 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2831d3-e0d4-4a4b-a209-d5cb09632279-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-v4ktn\" (UID: \"fd2831d3-e0d4-4a4b-a209-d5cb09632279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" Oct 31 00:00:09.951913 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:09.951900 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2831d3-e0d4-4a4b-a209-d5cb09632279-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-v4ktn\" (UID: \"fd2831d3-e0d4-4a4b-a209-d5cb09632279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" Oct 31 00:00:09.952178 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:09.951929 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd2831d3-e0d4-4a4b-a209-d5cb09632279-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-v4ktn\" (UID: \"fd2831d3-e0d4-4a4b-a209-d5cb09632279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" Oct 31 00:00:09.952178 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:09.951945 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2831d3-e0d4-4a4b-a209-d5cb09632279-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-v4ktn\" (UID: \"fd2831d3-e0d4-4a4b-a209-d5cb09632279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" Oct 31 00:00:09.952178 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:09.951973 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f28bj\" (UniqueName: \"kubernetes.io/projected/fd2831d3-e0d4-4a4b-a209-d5cb09632279-kube-api-access-f28bj\") pod \"maas-default-gateway-openshift-default-5df48fcc99-v4ktn\" (UID: \"fd2831d3-e0d4-4a4b-a209-d5cb09632279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" Oct 31 00:00:09.952396 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:09.952220 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2831d3-e0d4-4a4b-a209-d5cb09632279-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-v4ktn\" (UID: \"fd2831d3-e0d4-4a4b-a209-d5cb09632279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" Oct 31 00:00:09.952396 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:09.952316 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2831d3-e0d4-4a4b-a209-d5cb09632279-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-v4ktn\" (UID: \"fd2831d3-e0d4-4a4b-a209-d5cb09632279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" Oct 31 00:00:09.952396 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:09.952367 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2831d3-e0d4-4a4b-a209-d5cb09632279-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-v4ktn\" (UID: \"fd2831d3-e0d4-4a4b-a209-d5cb09632279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" Oct 31 00:00:09.952568 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:09.952509 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2831d3-e0d4-4a4b-a209-d5cb09632279-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-v4ktn\" (UID: \"fd2831d3-e0d4-4a4b-a209-d5cb09632279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" Oct 31 00:00:09.952628 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:09.952607 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2831d3-e0d4-4a4b-a209-d5cb09632279-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-v4ktn\" (UID: \"fd2831d3-e0d4-4a4b-a209-d5cb09632279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" Oct 31 00:00:09.954246 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:09.954230 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2831d3-e0d4-4a4b-a209-d5cb09632279-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-v4ktn\" (UID: \"fd2831d3-e0d4-4a4b-a209-d5cb09632279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" Oct 31 00:00:09.954619 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:09.954604 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2831d3-e0d4-4a4b-a209-d5cb09632279-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-v4ktn\" (UID: \"fd2831d3-e0d4-4a4b-a209-d5cb09632279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" Oct 31 00:00:10.009282 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:10.009228 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd2831d3-e0d4-4a4b-a209-d5cb09632279-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-v4ktn\" (UID: \"fd2831d3-e0d4-4a4b-a209-d5cb09632279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" Oct 31 00:00:10.009417 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:10.009399 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f28bj\" (UniqueName: \"kubernetes.io/projected/fd2831d3-e0d4-4a4b-a209-d5cb09632279-kube-api-access-f28bj\") pod \"maas-default-gateway-openshift-default-5df48fcc99-v4ktn\" (UID: \"fd2831d3-e0d4-4a4b-a209-d5cb09632279\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" Oct 31 00:00:10.044275 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:10.044253 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" Oct 31 00:00:10.468431 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:10.468405 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn"] Oct 31 00:00:10.469240 ip-10-0-130-177 kubenswrapper[4952]: W1031 00:00:10.469215 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd2831d3_e0d4_4a4b_a209_d5cb09632279.slice/crio-5f1f86682e0ef42e6e9377b8b3464a916a606f36b9b2197401bf5dfb1d2fbe53 WatchSource:0}: Error finding container 5f1f86682e0ef42e6e9377b8b3464a916a606f36b9b2197401bf5dfb1d2fbe53: Status 404 returned error can't find the container with id 5f1f86682e0ef42e6e9377b8b3464a916a606f36b9b2197401bf5dfb1d2fbe53 Oct 31 00:00:10.471239 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:10.471208 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:00:10.471305 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:10.471271 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:00:10.471348 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:10.471303 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:00:11.007644 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:11.007597 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" event={"ID":"fd2831d3-e0d4-4a4b-a209-d5cb09632279","Type":"ContainerStarted","Data":"0acd98d9cedba8e4725e45b8d7cc4bafd3bacdcd6358e7ca924f2224c9aa9ac4"} Oct 31 00:00:11.008153 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:11.007665 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" event={"ID":"fd2831d3-e0d4-4a4b-a209-d5cb09632279","Type":"ContainerStarted","Data":"5f1f86682e0ef42e6e9377b8b3464a916a606f36b9b2197401bf5dfb1d2fbe53"} Oct 31 00:00:11.045096 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:11.045065 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" Oct 31 00:00:11.046792 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:11.046766 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-v4ktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 00:00:11.046932 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:11.046822 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" podUID="fd2831d3-e0d4-4a4b-a209-d5cb09632279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 00:00:11.127169 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:11.127117 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" podStartSLOduration=2.127103428 podStartE2EDuration="2.127103428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:11.126785061 +0000 UTC m=+3230.270615278" watchObservedRunningTime="2026-10-31 00:00:11.127103428 +0000 UTC m=+3230.270933709" Oct 31 00:00:12.044782 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:12.044750 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-v4ktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 00:00:12.045222 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:12.044820 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" podUID="fd2831d3-e0d4-4a4b-a209-d5cb09632279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 00:00:13.045508 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:13.045424 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-v4ktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 00:00:13.045508 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:13.045484 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" podUID="fd2831d3-e0d4-4a4b-a209-d5cb09632279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 00:00:14.044740 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:14.044707 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-v4ktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 00:00:14.044947 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:14.044774 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" podUID="fd2831d3-e0d4-4a4b-a209-d5cb09632279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 00:00:15.045188 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:15.045137 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-v4ktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 00:00:15.045589 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:15.045207 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" podUID="fd2831d3-e0d4-4a4b-a209-d5cb09632279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 00:00:15.499284 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:15.499246 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:00:16.045514 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:16.045477 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-v4ktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 00:00:16.045928 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:16.045531 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" podUID="fd2831d3-e0d4-4a4b-a209-d5cb09632279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 00:00:17.044866 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:17.044830 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-v4ktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 00:00:17.045059 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:17.044904 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" podUID="fd2831d3-e0d4-4a4b-a209-d5cb09632279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 00:00:19.044702 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:19.044662 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-v4ktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:00:19.045401 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:19.045359 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" podUID="fd2831d3-e0d4-4a4b-a209-d5cb09632279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:00:20.088240 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:20.088196 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-v4ktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:00:20.088631 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:20.088277 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" podUID="fd2831d3-e0d4-4a4b-a209-d5cb09632279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:00:20.466969 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:20.466890 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" Oct 31 00:00:20.467280 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:20.467257 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" Oct 31 00:00:20.468063 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:20.468018 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" Oct 31 00:00:21.141367 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:00:21.141327 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:00:26.514111 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:26.514079 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:00:28.989816 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:28.989789 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:00:39.974988 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:39.974955 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq"] Oct 31 00:00:39.978726 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:39.978708 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" Oct 31 00:00:40.128503 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:40.128475 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6flbq\" (UID: \"3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" Oct 31 00:00:40.128503 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:40.128505 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6flbq\" (UID: \"3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" Oct 31 00:00:40.128773 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:40.128528 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8sg4\" (UniqueName: \"kubernetes.io/projected/3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f-kube-api-access-d8sg4\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6flbq\" (UID: \"3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" Oct 31 00:00:40.128773 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:40.128551 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6flbq\" (UID: \"3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" Oct 31 00:00:40.128773 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:40.128680 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6flbq\" (UID: \"3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" Oct 31 00:00:40.128773 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:40.128720 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6flbq\" (UID: \"3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" Oct 31 00:00:40.128773 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:40.128739 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6flbq\" (UID: \"3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" Oct 31 00:00:40.129079 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:40.128844 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6flbq\" (UID: \"3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" Oct 31 00:00:40.129079 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:40.128947 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6flbq\" (UID: \"3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" Oct 31 00:00:40.189510 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:40.189480 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq"] Oct 31 00:00:40.230126 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:40.230067 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6flbq\" (UID: \"3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" Oct 31 00:00:40.230126 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:40.230111 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6flbq\" (UID: \"3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" Oct 31 00:00:40.230294 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:40.230146 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6flbq\" (UID: \"3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" Oct 31 00:00:40.230380 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:40.230356 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6flbq\" (UID: \"3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" Oct 31 00:00:40.230419 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:40.230406 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6flbq\" (UID: \"3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" Oct 31 00:00:40.230480 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:40.230450 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8sg4\" (UniqueName: \"kubernetes.io/projected/3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f-kube-api-access-d8sg4\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6flbq\" (UID: \"3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" Oct 31 00:00:40.230533 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:40.230493 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6flbq\" (UID: \"3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" Oct 31 00:00:40.230587 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:40.230535 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6flbq\" (UID: \"3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" Oct 31 00:00:40.230587 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:40.230579 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6flbq\" (UID: \"3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" Oct 31 00:00:40.230691 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:40.230610 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6flbq\" (UID: \"3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" Oct 31 00:00:40.230751 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:40.230722 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6flbq\" (UID: \"3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" Oct 31 00:00:40.230889 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:40.230865 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6flbq\" (UID: \"3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" Oct 31 00:00:40.231174 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:40.231153 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6flbq\" (UID: \"3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" Oct 31 00:00:40.231228 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:40.231170 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6flbq\" (UID: \"3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" Oct 31 00:00:40.232800 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:40.232783 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6flbq\" (UID: \"3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" Oct 31 00:00:40.232882 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:40.232836 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6flbq\" (UID: \"3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" Oct 31 00:00:40.288050 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:40.288010 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8sg4\" (UniqueName: \"kubernetes.io/projected/3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f-kube-api-access-d8sg4\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6flbq\" (UID: \"3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" Oct 31 00:00:40.288222 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:40.288205 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6flbq\" (UID: \"3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" Oct 31 00:00:40.292147 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:40.292127 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" Oct 31 00:00:40.715366 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:40.715338 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq"] Oct 31 00:00:40.716876 ip-10-0-130-177 kubenswrapper[4952]: W1031 00:00:40.716849 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a6cdb7b_cb0d_44fd_a06e_607b947ddb3f.slice/crio-28d13285a76df4ac48e1e89606c7db6fb3ae701823c84c705fb09c8a2f3435c2 WatchSource:0}: Error finding container 28d13285a76df4ac48e1e89606c7db6fb3ae701823c84c705fb09c8a2f3435c2: Status 404 returned error can't find the container with id 28d13285a76df4ac48e1e89606c7db6fb3ae701823c84c705fb09c8a2f3435c2 Oct 31 00:00:40.718945 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:40.718914 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:00:40.719021 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:40.718982 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:00:40.719021 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:40.719013 4952 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 00:00:41.196999 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:41.196962 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" event={"ID":"3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f","Type":"ContainerStarted","Data":"43d144a653d17a68f6b6eba91c3d3f3c8422f184515267360aa6ce60be168497"} Oct 31 00:00:41.197507 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:41.197017 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" event={"ID":"3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f","Type":"ContainerStarted","Data":"28d13285a76df4ac48e1e89606c7db6fb3ae701823c84c705fb09c8a2f3435c2"} Oct 31 00:00:41.292338 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:41.292297 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" Oct 31 00:00:41.293916 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:41.293890 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-6flbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 31 00:00:41.294063 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:41.293937 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" podUID="3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 31 00:00:41.399523 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:41.399468 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" podStartSLOduration=2.399449926 podStartE2EDuration="2.399449926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:41.363677823 +0000 UTC m=+3260.507508053" watchObservedRunningTime="2026-10-31 00:00:41.399449926 +0000 UTC m=+3260.543280215" Oct 31 00:00:42.292805 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:42.292772 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-6flbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 31 00:00:42.293287 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:42.292819 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" podUID="3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 31 00:00:43.292472 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:43.292434 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-6flbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 31 00:00:43.292657 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:43.292487 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" podUID="3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 31 00:00:44.292880 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:44.292846 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-6flbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 31 00:00:44.293306 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:44.292897 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" podUID="3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 31 00:00:45.293268 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:45.293232 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-6flbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 31 00:00:45.293659 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:45.293299 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" podUID="3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 31 00:00:46.293513 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:46.293475 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-6flbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 31 00:00:46.294008 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:46.293568 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" podUID="3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 31 00:00:47.043091 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:47.043057 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:00:47.292848 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:47.292817 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-6flbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 31 00:00:47.293003 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:47.292873 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" podUID="3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 31 00:00:49.293872 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:49.293826 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-6flbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:00:49.294315 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:49.293881 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" podUID="3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:00:50.335180 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:50.335147 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-6flbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:00:50.335567 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:50.335200 4952 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" podUID="3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:00:50.932099 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:50.932062 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" Oct 31 00:00:50.932727 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:50.932700 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" Oct 31 00:00:50.933630 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:50.933612 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" Oct 31 00:00:51.071843 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:51.071814 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:00:51.141310 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:00:51.141279 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:00:51.294714 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:00:51.294640 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:01:04.002277 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:01:04.002190 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:01:04.525817 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:01:04.525778 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:01:06.538253 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:01:06.538204 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-v4ktn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:01:06.538915 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:01:06.538877 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-v4ktn" podUID="fd2831d3-e0d4-4a4b-a209-d5cb09632279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:01:21.142183 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:01:21.142143 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:01:21.489472 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:01:21.489396 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft_91651ce2-4673-4ced-b26d-e94ff8829d11/extract/0.log" Oct 31 00:01:21.501228 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:01:21.501196 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/2.log" Oct 31 00:01:21.509549 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:01:21.509522 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 31 00:01:21.523257 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:01:21.523238 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft_91651ce2-4673-4ced-b26d-e94ff8829d11/extract/0.log" Oct 31 00:01:21.533594 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:01:21.533577 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/2.log" Oct 31 00:01:21.539172 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:01:21.539156 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 31 00:01:29.330851 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:01:29.330816 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:01:35.118194 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:01:35.118151 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:01:45.696598 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:01:45.696559 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:01:51.142489 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:01:51.142452 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:01:58.345308 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:01:58.345270 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:02:21.141502 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:02:21.141466 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:02:23.591889 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:02:23.591855 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:02:29.758001 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:02:29.757968 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:33.249422 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:02:33.249382 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:02:38.047592 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:02:38.047564 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8ab4cea-mv7np"] Oct 31 00:02:38.097725 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:02:38.097695 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8ab4cea-mv7np"] Oct 31 00:02:38.974068 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:02:38.974016 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="314cf6d3-fdc9-4496-b4de-2e531820361c" path="/var/lib/kubelet/pods/314cf6d3-fdc9-4496-b4de-2e531820361c/volumes" Oct 31 00:02:40.070778 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:02:40.070744 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee80d40a-vszm6"] Oct 31 00:02:40.120253 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:02:40.120221 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee80d40a-vszm6"] Oct 31 00:02:40.968966 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:02:40.968934 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="789348bc-4698-479b-b2b0-da808059305f" path="/var/lib/kubelet/pods/789348bc-4698-479b-b2b0-da808059305f/volumes" Oct 31 00:02:51.142460 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:02:51.142422 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:02:51.600649 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:02:51.600623 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:03:03.070808 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:03:03.070777 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8608b102-8sl2j"] Oct 31 00:03:03.072979 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:03:03.072957 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8608b102-8sl2j"] Oct 31 00:03:04.224647 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:03:04.224612 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:03:04.968976 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:03:04.968942 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28642158-9697-4b4a-b1f2-55132b8eeeaf" path="/var/lib/kubelet/pods/28642158-9697-4b4a-b1f2-55132b8eeeaf/volumes" Oct 31 00:03:21.142205 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:03:21.142164 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:03:22.160367 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:03:22.160335 4952 scope.go:122] "RemoveContainer" containerID="96ed37579e3eeedb8b302ea720707284b0599dada7c1d5a158737fee400a0403" Oct 31 00:03:24.205152 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:03:24.205116 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:03:29.967709 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:03:29.967664 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:43.903566 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:03:43.903526 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:46.147163 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:03:46.147128 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:03:51.141847 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:03:51.141811 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:03:59.433604 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:03:59.433571 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:04:15.478806 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:04:15.478767 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:21.141084 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:04:21.141025 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:04:22.187480 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:04:22.187447 4952 scope.go:122] "RemoveContainer" containerID="6c4492230600cb02e2b3c38b8ff473851e2ce84687eb9f3d1c85c5eaa2edb756" Oct 31 00:04:22.197307 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:04:22.197288 4952 scope.go:122] "RemoveContainer" containerID="02cb6003ac30174ac5d59f0161caa5f8bb5e8235c46d44f21db98b4406ee6b52" Oct 31 00:04:22.206150 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:04:22.206127 4952 scope.go:122] "RemoveContainer" containerID="7534751ae85c1cdba3154aaac2c54ef545fb71d5278d11ac065db4cdf4d3e3e1" Oct 31 00:04:30.402217 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:04:30.402174 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:31.708794 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:04:31.708764 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:04:51.141644 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:04:51.141608 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:04:53.355047 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:04:53.355011 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:04:55.735230 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:04:55.735183 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-dz26x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:04:55.735828 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:04:55.735789 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-dz26x" podUID="f5421fcf-649c-477a-b487-68c52d23f09b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:05:04.037395 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:04.037361 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:05:07.052951 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:07.052918 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l8w2v"] Oct 31 00:05:07.102897 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:07.102872 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-l8w2v"] Oct 31 00:05:08.969008 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:08.968974 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a75e2636-1bbc-4dfd-a4a8-3a867026f118" path="/var/lib/kubelet/pods/a75e2636-1bbc-4dfd-a4a8-3a867026f118/volumes" Oct 31 00:05:21.141494 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:05:21.141449 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:05:24.095986 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:24.095949 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:24.666920 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:24.666889 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:25.089640 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:25.089609 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:29.257424 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:29.257389 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:29.504527 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:29.504489 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:30.724940 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:30.724864 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:32.546013 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:32.545975 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:32.783387 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:32.783356 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:35.703669 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:35.703603 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:35.941441 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:35.941396 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:40.668234 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:40.668200 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:43.718248 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:43.718211 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:05:45.613690 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:45.613661 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:50.478301 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:50.478240 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:51.142127 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:05:51.142089 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:05:54.035844 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:54.035796 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:54.600823 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:54.600794 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:54.900028 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:54.899939 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:55.797783 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:55.797740 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:57.748389 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:57.748355 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n"] Oct 31 00:05:57.752153 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:57.752127 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n" Oct 31 00:05:57.805016 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:57.804987 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-9b1704\"/\"openshift-service-ca.crt\"" Oct 31 00:05:57.857270 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:57.857229 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-9b1704\"/\"simulator-single-main-9b1704-kserve-self-signed-certs\"" Oct 31 00:05:57.857510 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:57.857297 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-9b1704\"/\"default-dockercfg-pkcfh\"" Oct 31 00:05:57.859796 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:57.859765 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n"] Oct 31 00:05:57.881851 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:57.881815 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72bb459f-fed9-4afd-ba77-a3bdffe1d680-home\") pod \"simulator-single-main-9b1704-kserve-96f489697-gdn9n\" (UID: \"72bb459f-fed9-4afd-ba77-a3bdffe1d680\") " pod="e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n" Oct 31 00:05:57.882071 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:57.881860 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72bb459f-fed9-4afd-ba77-a3bdffe1d680-dshm\") pod \"simulator-single-main-9b1704-kserve-96f489697-gdn9n\" (UID: \"72bb459f-fed9-4afd-ba77-a3bdffe1d680\") " pod="e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n" Oct 31 00:05:57.882071 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:57.881899 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72bb459f-fed9-4afd-ba77-a3bdffe1d680-model-cache\") pod \"simulator-single-main-9b1704-kserve-96f489697-gdn9n\" (UID: \"72bb459f-fed9-4afd-ba77-a3bdffe1d680\") " pod="e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n" Oct 31 00:05:57.882071 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:57.881978 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72bb459f-fed9-4afd-ba77-a3bdffe1d680-tmp-dir\") pod \"simulator-single-main-9b1704-kserve-96f489697-gdn9n\" (UID: \"72bb459f-fed9-4afd-ba77-a3bdffe1d680\") " pod="e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n" Oct 31 00:05:57.882071 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:57.882055 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72bb459f-fed9-4afd-ba77-a3bdffe1d680-tls-certs\") pod \"simulator-single-main-9b1704-kserve-96f489697-gdn9n\" (UID: \"72bb459f-fed9-4afd-ba77-a3bdffe1d680\") " pod="e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n" Oct 31 00:05:57.882228 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:57.882088 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm6ls\" (UniqueName: \"kubernetes.io/projected/72bb459f-fed9-4afd-ba77-a3bdffe1d680-kube-api-access-dm6ls\") pod \"simulator-single-main-9b1704-kserve-96f489697-gdn9n\" (UID: \"72bb459f-fed9-4afd-ba77-a3bdffe1d680\") " pod="e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n" Oct 31 00:05:57.905324 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:57.905288 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-9b1704\"/\"kube-root-ca.crt\"" Oct 31 00:05:57.935300 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:57.935232 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:57.982817 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:57.982784 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm6ls\" (UniqueName: \"kubernetes.io/projected/72bb459f-fed9-4afd-ba77-a3bdffe1d680-kube-api-access-dm6ls\") pod \"simulator-single-main-9b1704-kserve-96f489697-gdn9n\" (UID: \"72bb459f-fed9-4afd-ba77-a3bdffe1d680\") " pod="e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n" Oct 31 00:05:57.983011 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:57.982893 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72bb459f-fed9-4afd-ba77-a3bdffe1d680-home\") pod \"simulator-single-main-9b1704-kserve-96f489697-gdn9n\" (UID: \"72bb459f-fed9-4afd-ba77-a3bdffe1d680\") " pod="e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n" Oct 31 00:05:57.983011 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:57.982931 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72bb459f-fed9-4afd-ba77-a3bdffe1d680-dshm\") pod \"simulator-single-main-9b1704-kserve-96f489697-gdn9n\" (UID: \"72bb459f-fed9-4afd-ba77-a3bdffe1d680\") " pod="e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n" Oct 31 00:05:57.983011 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:57.982969 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72bb459f-fed9-4afd-ba77-a3bdffe1d680-model-cache\") pod \"simulator-single-main-9b1704-kserve-96f489697-gdn9n\" (UID: \"72bb459f-fed9-4afd-ba77-a3bdffe1d680\") " pod="e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n" Oct 31 00:05:57.983011 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:57.982992 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72bb459f-fed9-4afd-ba77-a3bdffe1d680-tmp-dir\") pod \"simulator-single-main-9b1704-kserve-96f489697-gdn9n\" (UID: \"72bb459f-fed9-4afd-ba77-a3bdffe1d680\") " pod="e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n" Oct 31 00:05:57.983249 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:57.983022 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72bb459f-fed9-4afd-ba77-a3bdffe1d680-tls-certs\") pod \"simulator-single-main-9b1704-kserve-96f489697-gdn9n\" (UID: \"72bb459f-fed9-4afd-ba77-a3bdffe1d680\") " pod="e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n" Oct 31 00:05:57.983461 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:57.983436 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72bb459f-fed9-4afd-ba77-a3bdffe1d680-model-cache\") pod \"simulator-single-main-9b1704-kserve-96f489697-gdn9n\" (UID: \"72bb459f-fed9-4afd-ba77-a3bdffe1d680\") " pod="e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n" Oct 31 00:05:57.983461 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:57.983454 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72bb459f-fed9-4afd-ba77-a3bdffe1d680-tmp-dir\") pod \"simulator-single-main-9b1704-kserve-96f489697-gdn9n\" (UID: \"72bb459f-fed9-4afd-ba77-a3bdffe1d680\") " pod="e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n" Oct 31 00:05:57.983604 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:57.983496 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72bb459f-fed9-4afd-ba77-a3bdffe1d680-home\") pod \"simulator-single-main-9b1704-kserve-96f489697-gdn9n\" (UID: \"72bb459f-fed9-4afd-ba77-a3bdffe1d680\") " pod="e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n" Oct 31 00:05:57.985501 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:57.985481 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72bb459f-fed9-4afd-ba77-a3bdffe1d680-dshm\") pod \"simulator-single-main-9b1704-kserve-96f489697-gdn9n\" (UID: \"72bb459f-fed9-4afd-ba77-a3bdffe1d680\") " pod="e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n" Oct 31 00:05:57.985671 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:57.985653 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72bb459f-fed9-4afd-ba77-a3bdffe1d680-tls-certs\") pod \"simulator-single-main-9b1704-kserve-96f489697-gdn9n\" (UID: \"72bb459f-fed9-4afd-ba77-a3bdffe1d680\") " pod="e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n" Oct 31 00:05:58.040687 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:58.040615 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm6ls\" (UniqueName: \"kubernetes.io/projected/72bb459f-fed9-4afd-ba77-a3bdffe1d680-kube-api-access-dm6ls\") pod \"simulator-single-main-9b1704-kserve-96f489697-gdn9n\" (UID: \"72bb459f-fed9-4afd-ba77-a3bdffe1d680\") " pod="e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n" Oct 31 00:05:58.064590 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:58.064561 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n" Oct 31 00:05:58.442129 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:58.442102 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n"] Oct 31 00:05:58.443586 ip-10-0-130-177 kubenswrapper[4952]: W1031 00:05:58.443558 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72bb459f_fed9_4afd_ba77_a3bdffe1d680.slice/crio-5a73402097a68d4e9b6bb5d92d2cdeac7046380e910375eb72a3460e33d1f551 WatchSource:0}: Error finding container 5a73402097a68d4e9b6bb5d92d2cdeac7046380e910375eb72a3460e33d1f551: Status 404 returned error can't find the container with id 5a73402097a68d4e9b6bb5d92d2cdeac7046380e910375eb72a3460e33d1f551 Oct 31 00:05:59.154390 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:59.154345 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n" event={"ID":"72bb459f-fed9-4afd-ba77-a3bdffe1d680","Type":"ContainerStarted","Data":"fd1e8c66120cfd4131ce30295611cf1ba50a297f2a295900584dcb8da1c2d5af"} Oct 31 00:05:59.154859 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:59.154401 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n" event={"ID":"72bb459f-fed9-4afd-ba77-a3bdffe1d680","Type":"ContainerStarted","Data":"5a73402097a68d4e9b6bb5d92d2cdeac7046380e910375eb72a3460e33d1f551"} Oct 31 00:05:59.270518 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:05:59.270454 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n" podStartSLOduration=2.270435837 podStartE2EDuration="2.270435837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:05:59.269409364 +0000 UTC m=+3578.413239582" watchObservedRunningTime="2026-10-31 00:05:59.270435837 +0000 UTC m=+3578.414266041" Oct 31 00:06:00.065732 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:06:00.065684 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n" Oct 31 00:06:00.662012 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:06:00.661976 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:02.421803 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:06:02.421770 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:06:05.488595 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:06:05.488561 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:08.065009 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:06:08.064979 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n" Oct 31 00:06:08.077908 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:06:08.077887 4952 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n" Oct 31 00:06:08.219767 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:06:08.219741 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n" Oct 31 00:06:10.266279 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:06:10.266244 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:15.167632 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:06:15.167600 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:16.254026 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:06:16.253990 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:16.476388 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:06:16.476353 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:19.964442 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:06:19.964407 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:21.142178 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:06:21.142133 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:06:21.535707 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:06:21.535607 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft_91651ce2-4673-4ced-b26d-e94ff8829d11/extract/0.log" Oct 31 00:06:21.553194 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:06:21.553161 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/2.log" Oct 31 00:06:21.558965 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:06:21.558933 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 31 00:06:21.566815 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:06:21.566790 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft_91651ce2-4673-4ced-b26d-e94ff8829d11/extract/0.log" Oct 31 00:06:21.577806 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:06:21.577784 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/2.log" Oct 31 00:06:21.583430 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:06:21.583411 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 31 00:06:22.243679 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:06:22.243647 4952 scope.go:122] "RemoveContainer" containerID="507ee0e6f988178e5662b6fdc5508f6dd69902f9204bce5a593ffad95c5e1133" Oct 31 00:06:24.863347 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:06:24.863309 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:27.808054 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:06:27.808011 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:06:29.653164 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:06:29.653122 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:31.066704 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:06:31.066668 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:34.458318 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:06:34.458278 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:39.261964 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:06:39.261927 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:44.192867 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:06:44.192833 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:48.963717 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:06:48.963676 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:51.142312 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:06:51.142267 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:06:53.567932 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:06:53.567897 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:53.803222 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:06:53.803185 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:56.679641 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:06:56.679610 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:06:58.404063 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:06:58.404002 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n"] Oct 31 00:06:58.556224 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:06:58.556194 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:58.769425 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:06:58.769338 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:59.057775 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:06:59.057681 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:03.954693 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:03.954611 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:05.956286 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:05.956247 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:06.169365 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:06.169328 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:08.657925 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:08.657886 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:12.888604 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:12.888565 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:13.424750 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:13.424686 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n" podUID="72bb459f-fed9-4afd-ba77-a3bdffe1d680" containerName="main" containerID="cri-o://fd1e8c66120cfd4131ce30295611cf1ba50a297f2a295900584dcb8da1c2d5af" gracePeriod=45 Oct 31 00:07:13.578885 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:13.578849 4952 generic.go:356] "Generic (PLEG): container finished" podID="72bb459f-fed9-4afd-ba77-a3bdffe1d680" containerID="fd1e8c66120cfd4131ce30295611cf1ba50a297f2a295900584dcb8da1c2d5af" exitCode=0 Oct 31 00:07:13.579071 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:13.578923 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n" event={"ID":"72bb459f-fed9-4afd-ba77-a3bdffe1d680","Type":"ContainerDied","Data":"fd1e8c66120cfd4131ce30295611cf1ba50a297f2a295900584dcb8da1c2d5af"} Oct 31 00:07:13.680734 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:13.680685 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n" Oct 31 00:07:13.736200 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:13.736174 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72bb459f-fed9-4afd-ba77-a3bdffe1d680-kube-api-access-dm6ls\" (UniqueName: \"kubernetes.io/projected/72bb459f-fed9-4afd-ba77-a3bdffe1d680-kube-api-access-dm6ls\") pod \"72bb459f-fed9-4afd-ba77-a3bdffe1d680\" (UID: \"72bb459f-fed9-4afd-ba77-a3bdffe1d680\") " Oct 31 00:07:13.736315 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:13.736272 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72bb459f-fed9-4afd-ba77-a3bdffe1d680-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72bb459f-fed9-4afd-ba77-a3bdffe1d680-tmp-dir\") pod \"72bb459f-fed9-4afd-ba77-a3bdffe1d680\" (UID: \"72bb459f-fed9-4afd-ba77-a3bdffe1d680\") " Oct 31 00:07:13.736356 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:13.736313 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72bb459f-fed9-4afd-ba77-a3bdffe1d680-home\" (UniqueName: \"kubernetes.io/empty-dir/72bb459f-fed9-4afd-ba77-a3bdffe1d680-home\") pod \"72bb459f-fed9-4afd-ba77-a3bdffe1d680\" (UID: \"72bb459f-fed9-4afd-ba77-a3bdffe1d680\") " Oct 31 00:07:13.736396 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:13.736356 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72bb459f-fed9-4afd-ba77-a3bdffe1d680-dshm\" (UniqueName: \"kubernetes.io/empty-dir/72bb459f-fed9-4afd-ba77-a3bdffe1d680-dshm\") pod \"72bb459f-fed9-4afd-ba77-a3bdffe1d680\" (UID: \"72bb459f-fed9-4afd-ba77-a3bdffe1d680\") " Oct 31 00:07:13.736429 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:13.736390 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72bb459f-fed9-4afd-ba77-a3bdffe1d680-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72bb459f-fed9-4afd-ba77-a3bdffe1d680-model-cache\") pod \"72bb459f-fed9-4afd-ba77-a3bdffe1d680\" (UID: \"72bb459f-fed9-4afd-ba77-a3bdffe1d680\") " Oct 31 00:07:13.736540 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:13.736515 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72bb459f-fed9-4afd-ba77-a3bdffe1d680-tls-certs\" (UniqueName: \"kubernetes.io/secret/72bb459f-fed9-4afd-ba77-a3bdffe1d680-tls-certs\") pod \"72bb459f-fed9-4afd-ba77-a3bdffe1d680\" (UID: \"72bb459f-fed9-4afd-ba77-a3bdffe1d680\") " Oct 31 00:07:13.736616 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:13.736557 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72bb459f-fed9-4afd-ba77-a3bdffe1d680-tmp-dir" pod "72bb459f-fed9-4afd-ba77-a3bdffe1d680" (UID: "72bb459f-fed9-4afd-ba77-a3bdffe1d680"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:07:13.736844 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:13.736630 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72bb459f-fed9-4afd-ba77-a3bdffe1d680-home" pod "72bb459f-fed9-4afd-ba77-a3bdffe1d680" (UID: "72bb459f-fed9-4afd-ba77-a3bdffe1d680"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:07:13.736844 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:13.736694 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72bb459f-fed9-4afd-ba77-a3bdffe1d680-model-cache" pod "72bb459f-fed9-4afd-ba77-a3bdffe1d680" (UID: "72bb459f-fed9-4afd-ba77-a3bdffe1d680"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:07:13.736995 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:13.736844 4952 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72bb459f-fed9-4afd-ba77-a3bdffe1d680-tmp-dir\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:07:13.738685 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:13.738651 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72bb459f-fed9-4afd-ba77-a3bdffe1d680-dshm" pod "72bb459f-fed9-4afd-ba77-a3bdffe1d680" (UID: "72bb459f-fed9-4afd-ba77-a3bdffe1d680"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:07:13.738767 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:13.738690 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72bb459f-fed9-4afd-ba77-a3bdffe1d680-tls-certs" pod "72bb459f-fed9-4afd-ba77-a3bdffe1d680" (UID: "72bb459f-fed9-4afd-ba77-a3bdffe1d680"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:07:13.738767 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:13.738714 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72bb459f-fed9-4afd-ba77-a3bdffe1d680-kube-api-access-dm6ls" pod "72bb459f-fed9-4afd-ba77-a3bdffe1d680" (UID: "72bb459f-fed9-4afd-ba77-a3bdffe1d680"). InnerVolumeSpecName "kube-api-access-dm6ls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:07:13.837428 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:13.837399 4952 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72bb459f-fed9-4afd-ba77-a3bdffe1d680-model-cache\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:07:13.837428 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:13.837426 4952 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72bb459f-fed9-4afd-ba77-a3bdffe1d680-tls-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:07:13.837620 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:13.837437 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm6ls\" (UniqueName: \"kubernetes.io/projected/72bb459f-fed9-4afd-ba77-a3bdffe1d680-kube-api-access-dm6ls\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:07:13.837620 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:13.837447 4952 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72bb459f-fed9-4afd-ba77-a3bdffe1d680-home\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:07:13.837620 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:13.837455 4952 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72bb459f-fed9-4afd-ba77-a3bdffe1d680-dshm\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:07:14.428098 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:14.428020 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:14.586644 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:14.586612 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n" Oct 31 00:07:14.586813 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:14.586650 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n" event={"ID":"72bb459f-fed9-4afd-ba77-a3bdffe1d680","Type":"ContainerDied","Data":"5a73402097a68d4e9b6bb5d92d2cdeac7046380e910375eb72a3460e33d1f551"} Oct 31 00:07:14.586813 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:14.586692 4952 scope.go:122] "RemoveContainer" containerID="fd1e8c66120cfd4131ce30295611cf1ba50a297f2a295900584dcb8da1c2d5af" Oct 31 00:07:14.753206 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:14.753116 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:14.754179 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:14.754154 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n"] Oct 31 00:07:14.803786 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:14.803759 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-9b1704/simulator-single-main-9b1704-kserve-96f489697-gdn9n"] Oct 31 00:07:14.969328 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:14.969301 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72bb459f-fed9-4afd-ba77-a3bdffe1d680" path="/var/lib/kubelet/pods/72bb459f-fed9-4afd-ba77-a3bdffe1d680/volumes" Oct 31 00:07:15.491860 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:15.491822 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:17.531601 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:17.531567 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:07:21.142494 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:07:21.142458 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:07:21.177462 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:21.177388 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:21.408113 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:21.408003 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:21.631405 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:21.631368 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:31.458344 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:31.458298 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:35.440315 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:35.440272 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:35.656212 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:35.656173 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:41.080505 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:07:41.080459 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:51.142420 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:07:51.142384 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:08:21.140442 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:08:21.140400 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:08:31.751996 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:08:31.751918 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:51.141501 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:08:51.141463 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:09:08.695602 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:09:08.695562 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:09:21.141219 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:09:21.141180 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:09:25.825615 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:09:25.825576 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:09:27.735146 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:09:27.735112 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:27.954519 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:09:27.954441 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:32.163088 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:09:32.163025 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:35.241697 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:09:35.241666 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:09:51.140646 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:09:51.140613 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:09:54.316541 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:09:54.316509 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:54.830150 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:09:54.830116 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:09:54.909333 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:09:54.909296 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:55.427932 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:09:55.427902 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:59.664390 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:09:59.664335 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:59.927805 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:09:59.927709 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:02.913636 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:10:02.913602 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:03.131527 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:10:03.131494 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:05.807574 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:10:05.807531 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:06.023487 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:10:06.023450 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:10.758829 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:10:10.758797 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:13.629694 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:10:13.629653 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:10:15.659184 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:10:15.659148 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:20.492083 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:10:20.492021 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:21.141965 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:10:21.141923 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:10:24.317109 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:10:24.317031 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:24.966887 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:10:24.966852 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:25.204136 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:10:25.204091 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:25.610296 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:10:25.610257 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:27.867895 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:10:27.867859 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:30.533252 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:10:30.533217 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:32.335279 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:10:32.335224 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:10:32.495544 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:10:32.495503 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:35.473388 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:10:35.473351 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:40.268703 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:10:40.268660 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:45.167220 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:10:45.167178 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:49.971321 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:10:49.971285 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:51.143108 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:10:51.143068 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:10:54.863545 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:10:54.863509 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:57.923605 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:10:57.923572 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:10:59.977850 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:10:59.977810 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:02.838212 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:11:02.838185 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:03.074693 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:11:03.074647 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:04.473423 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:11:04.473390 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:09.264880 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:11:09.264848 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:14.083370 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:11:14.083319 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:18.857399 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:11:18.857354 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:21.142768 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:11:21.142728 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:11:21.583691 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:11:21.583622 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft_91651ce2-4673-4ced-b26d-e94ff8829d11/extract/0.log" Oct 31 00:11:21.593881 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:11:21.593860 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/2.log" Oct 31 00:11:21.599666 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:11:21.599646 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 31 00:11:21.610809 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:11:21.610791 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft_91651ce2-4673-4ced-b26d-e94ff8829d11/extract/0.log" Oct 31 00:11:21.620933 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:11:21.620918 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/2.log" Oct 31 00:11:21.626590 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:11:21.626573 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 31 00:11:23.749896 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:11:23.749856 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:24.026465 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:11:24.026367 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:25.874838 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:11:25.874799 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:11:28.987392 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:11:28.987361 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:29.205115 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:11:29.205079 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:32.838283 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:11:32.838194 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:33.767459 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:11:33.767429 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:34.104988 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:11:34.104959 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:36.359621 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:11:36.359587 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:36.585958 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:11:36.585924 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:38.471064 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:11:38.470998 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:42.731154 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:11:42.731098 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:44.251541 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:11:44.251510 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:44.640389 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:11:44.640359 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:45.305568 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:11:45.305536 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:47.314593 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:11:47.314559 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:11:51.141595 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:11:51.141538 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:11:51.590444 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:11:51.590404 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:51.880087 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:11:51.879982 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:52.139613 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:11:52.139524 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:59.550308 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:11:59.550272 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:04.102960 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:12:04.102918 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:12:06.986193 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:12:06.986158 4952 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-6flbq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:12:06.986796 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:12:06.986763 4952 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6flbq" podUID="3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:12:14.306645 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:12:14.306604 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:14.529779 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:12:14.529746 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:21.142868 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:12:21.142830 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:12:29.841145 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:12:29.841111 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:34.248170 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:12:34.248131 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:42.325257 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:12:42.325214 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:45.113686 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:12:45.113642 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:12:51.142111 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:12:51.142072 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:13:14.725087 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:13:14.725052 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:13:21.141539 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:13:21.141500 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:13:26.289660 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:13:26.289631 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:30.019720 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:13:30.019686 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:13:32.996064 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:13:32.990962 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:33.529677 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:13:33.529634 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:45.227486 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:13:45.227450 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:13:51.141080 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:13:51.141019 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:13:56.926190 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:13:56.926153 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:14:05.678099 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:05.678065 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:19.510885 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:19.510846 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:14:19.815629 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:19.815557 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-84b8bf64d7-djfcl"] Oct 31 00:14:19.815961 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:19.815943 4952 cpu_manager.go:414] "containerMap: removing container" podUID="72bb459f-fed9-4afd-ba77-a3bdffe1d680" containerName="main" Oct 31 00:14:19.815961 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:19.815957 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72bb459f-fed9-4afd-ba77-a3bdffe1d680" containerName="main" Oct 31 00:14:19.816073 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:19.816027 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="72bb459f-fed9-4afd-ba77-a3bdffe1d680" containerName="main" Oct 31 00:14:19.819251 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:19.819236 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84b8bf64d7-djfcl" Oct 31 00:14:19.919481 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:19.919442 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ae58c73-211f-4a73-b0f7-8e1d5c092b1a-maas-api-tls\") pod \"maas-api-84b8bf64d7-djfcl\" (UID: \"7ae58c73-211f-4a73-b0f7-8e1d5c092b1a\") " pod="odh-ai-gateway-infra/maas-api-84b8bf64d7-djfcl" Oct 31 00:14:19.919667 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:19.919561 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj6mj\" (UniqueName: \"kubernetes.io/projected/7ae58c73-211f-4a73-b0f7-8e1d5c092b1a-kube-api-access-vj6mj\") pod \"maas-api-84b8bf64d7-djfcl\" (UID: \"7ae58c73-211f-4a73-b0f7-8e1d5c092b1a\") " pod="odh-ai-gateway-infra/maas-api-84b8bf64d7-djfcl" Oct 31 00:14:19.919667 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:19.919607 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ae58c73-211f-4a73-b0f7-8e1d5c092b1a-metrics-certs\") pod \"maas-api-84b8bf64d7-djfcl\" (UID: \"7ae58c73-211f-4a73-b0f7-8e1d5c092b1a\") " pod="odh-ai-gateway-infra/maas-api-84b8bf64d7-djfcl" Oct 31 00:14:19.919667 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:19.919636 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ae58c73-211f-4a73-b0f7-8e1d5c092b1a-openshift-service-ca\") pod \"maas-api-84b8bf64d7-djfcl\" (UID: \"7ae58c73-211f-4a73-b0f7-8e1d5c092b1a\") " pod="odh-ai-gateway-infra/maas-api-84b8bf64d7-djfcl" Oct 31 00:14:19.922706 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:19.922676 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s5prt\"" Oct 31 00:14:19.922706 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:19.922676 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 00:14:19.922906 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:19.922726 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 00:14:19.926986 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:19.926957 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84b8bf64d7-djfcl"] Oct 31 00:14:20.021174 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:20.021145 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ae58c73-211f-4a73-b0f7-8e1d5c092b1a-maas-api-tls\") pod \"maas-api-84b8bf64d7-djfcl\" (UID: \"7ae58c73-211f-4a73-b0f7-8e1d5c092b1a\") " pod="odh-ai-gateway-infra/maas-api-84b8bf64d7-djfcl" Oct 31 00:14:20.021367 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:20.021229 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj6mj\" (UniqueName: \"kubernetes.io/projected/7ae58c73-211f-4a73-b0f7-8e1d5c092b1a-kube-api-access-vj6mj\") pod \"maas-api-84b8bf64d7-djfcl\" (UID: \"7ae58c73-211f-4a73-b0f7-8e1d5c092b1a\") " pod="odh-ai-gateway-infra/maas-api-84b8bf64d7-djfcl" Oct 31 00:14:20.021367 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:20.021268 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ae58c73-211f-4a73-b0f7-8e1d5c092b1a-metrics-certs\") pod \"maas-api-84b8bf64d7-djfcl\" (UID: \"7ae58c73-211f-4a73-b0f7-8e1d5c092b1a\") " pod="odh-ai-gateway-infra/maas-api-84b8bf64d7-djfcl" Oct 31 00:14:20.021367 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:20.021298 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ae58c73-211f-4a73-b0f7-8e1d5c092b1a-openshift-service-ca\") pod \"maas-api-84b8bf64d7-djfcl\" (UID: \"7ae58c73-211f-4a73-b0f7-8e1d5c092b1a\") " pod="odh-ai-gateway-infra/maas-api-84b8bf64d7-djfcl" Oct 31 00:14:20.022304 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:20.022279 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ae58c73-211f-4a73-b0f7-8e1d5c092b1a-openshift-service-ca\") pod \"maas-api-84b8bf64d7-djfcl\" (UID: \"7ae58c73-211f-4a73-b0f7-8e1d5c092b1a\") " pod="odh-ai-gateway-infra/maas-api-84b8bf64d7-djfcl" Oct 31 00:14:20.024098 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:20.024071 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ae58c73-211f-4a73-b0f7-8e1d5c092b1a-metrics-certs\") pod \"maas-api-84b8bf64d7-djfcl\" (UID: \"7ae58c73-211f-4a73-b0f7-8e1d5c092b1a\") " pod="odh-ai-gateway-infra/maas-api-84b8bf64d7-djfcl" Oct 31 00:14:20.024206 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:20.024136 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ae58c73-211f-4a73-b0f7-8e1d5c092b1a-maas-api-tls\") pod \"maas-api-84b8bf64d7-djfcl\" (UID: \"7ae58c73-211f-4a73-b0f7-8e1d5c092b1a\") " pod="odh-ai-gateway-infra/maas-api-84b8bf64d7-djfcl" Oct 31 00:14:20.077822 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:20.077765 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj6mj\" (UniqueName: \"kubernetes.io/projected/7ae58c73-211f-4a73-b0f7-8e1d5c092b1a-kube-api-access-vj6mj\") pod \"maas-api-84b8bf64d7-djfcl\" (UID: \"7ae58c73-211f-4a73-b0f7-8e1d5c092b1a\") " pod="odh-ai-gateway-infra/maas-api-84b8bf64d7-djfcl" Oct 31 00:14:20.132954 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:20.132931 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84b8bf64d7-djfcl" Oct 31 00:14:20.503231 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:20.503208 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84b8bf64d7-djfcl"] Oct 31 00:14:20.504860 ip-10-0-130-177 kubenswrapper[4952]: W1031 00:14:20.504833 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae58c73_211f_4a73_b0f7_8e1d5c092b1a.slice/crio-683abc5482517a00abd61f15765a1a221313cb722c39cd80a5cf4dfbc436f099 WatchSource:0}: Error finding container 683abc5482517a00abd61f15765a1a221313cb722c39cd80a5cf4dfbc436f099: Status 404 returned error can't find the container with id 683abc5482517a00abd61f15765a1a221313cb722c39cd80a5cf4dfbc436f099 Oct 31 00:14:20.506280 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:20.506262 4952 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:14:21.092071 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:21.092021 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84b8bf64d7-djfcl" event={"ID":"7ae58c73-211f-4a73-b0f7-8e1d5c092b1a","Type":"ContainerStarted","Data":"683abc5482517a00abd61f15765a1a221313cb722c39cd80a5cf4dfbc436f099"} Oct 31 00:14:21.140662 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:14:21.140628 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:14:22.098952 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:22.098918 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84b8bf64d7-djfcl" event={"ID":"7ae58c73-211f-4a73-b0f7-8e1d5c092b1a","Type":"ContainerStarted","Data":"846ffd94c3debb6227bb5894c9c71aebd7af187fea0ca708efd969687c021c67"} Oct 31 00:14:22.099397 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:22.099066 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-84b8bf64d7-djfcl" Oct 31 00:14:22.217486 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:22.217441 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-84b8bf64d7-djfcl" podStartSLOduration=2.40622884 podStartE2EDuration="3.217428011s" totalImagesPullingTime="811.199171ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:22.21598814 +0000 UTC m=+4081.359818344" watchObservedRunningTime="2026-10-31 00:14:22.217428011 +0000 UTC m=+4081.361258247" Oct 31 00:14:26.454180 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:26.454140 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:27.086144 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:27.086102 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:27.358803 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:27.358727 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:28.109020 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:28.108994 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-84b8bf64d7-djfcl" Oct 31 00:14:28.307993 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:28.307959 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-85b64959d7-vwq5c"] Oct 31 00:14:28.311934 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:28.311915 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85b64959d7-vwq5c" Oct 31 00:14:28.385386 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:28.385303 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-85b64959d7-vwq5c"] Oct 31 00:14:28.390472 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:28.390449 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd2d26d1-c967-40d3-a5e4-9633bf2c4ed5-metrics-certs\") pod \"maas-api-85b64959d7-vwq5c\" (UID: \"cd2d26d1-c967-40d3-a5e4-9633bf2c4ed5\") " pod="odh-ai-gateway-infra/maas-api-85b64959d7-vwq5c" Oct 31 00:14:28.390600 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:28.390523 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd2d26d1-c967-40d3-a5e4-9633bf2c4ed5-maas-api-tls\") pod \"maas-api-85b64959d7-vwq5c\" (UID: \"cd2d26d1-c967-40d3-a5e4-9633bf2c4ed5\") " pod="odh-ai-gateway-infra/maas-api-85b64959d7-vwq5c" Oct 31 00:14:28.390600 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:28.390562 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrjw9\" (UniqueName: \"kubernetes.io/projected/cd2d26d1-c967-40d3-a5e4-9633bf2c4ed5-kube-api-access-jrjw9\") pod \"maas-api-85b64959d7-vwq5c\" (UID: \"cd2d26d1-c967-40d3-a5e4-9633bf2c4ed5\") " pod="odh-ai-gateway-infra/maas-api-85b64959d7-vwq5c" Oct 31 00:14:28.390679 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:28.390603 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd2d26d1-c967-40d3-a5e4-9633bf2c4ed5-openshift-service-ca\") pod \"maas-api-85b64959d7-vwq5c\" (UID: \"cd2d26d1-c967-40d3-a5e4-9633bf2c4ed5\") " pod="odh-ai-gateway-infra/maas-api-85b64959d7-vwq5c" Oct 31 00:14:28.491164 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:28.491135 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd2d26d1-c967-40d3-a5e4-9633bf2c4ed5-openshift-service-ca\") pod \"maas-api-85b64959d7-vwq5c\" (UID: \"cd2d26d1-c967-40d3-a5e4-9633bf2c4ed5\") " pod="odh-ai-gateway-infra/maas-api-85b64959d7-vwq5c" Oct 31 00:14:28.491342 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:28.491203 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd2d26d1-c967-40d3-a5e4-9633bf2c4ed5-metrics-certs\") pod \"maas-api-85b64959d7-vwq5c\" (UID: \"cd2d26d1-c967-40d3-a5e4-9633bf2c4ed5\") " pod="odh-ai-gateway-infra/maas-api-85b64959d7-vwq5c" Oct 31 00:14:28.491342 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:28.491265 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd2d26d1-c967-40d3-a5e4-9633bf2c4ed5-maas-api-tls\") pod \"maas-api-85b64959d7-vwq5c\" (UID: \"cd2d26d1-c967-40d3-a5e4-9633bf2c4ed5\") " pod="odh-ai-gateway-infra/maas-api-85b64959d7-vwq5c" Oct 31 00:14:28.491342 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:28.491291 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrjw9\" (UniqueName: \"kubernetes.io/projected/cd2d26d1-c967-40d3-a5e4-9633bf2c4ed5-kube-api-access-jrjw9\") pod \"maas-api-85b64959d7-vwq5c\" (UID: \"cd2d26d1-c967-40d3-a5e4-9633bf2c4ed5\") " pod="odh-ai-gateway-infra/maas-api-85b64959d7-vwq5c" Oct 31 00:14:28.491798 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:28.491770 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd2d26d1-c967-40d3-a5e4-9633bf2c4ed5-openshift-service-ca\") pod \"maas-api-85b64959d7-vwq5c\" (UID: \"cd2d26d1-c967-40d3-a5e4-9633bf2c4ed5\") " pod="odh-ai-gateway-infra/maas-api-85b64959d7-vwq5c" Oct 31 00:14:28.493947 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:28.493921 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd2d26d1-c967-40d3-a5e4-9633bf2c4ed5-metrics-certs\") pod \"maas-api-85b64959d7-vwq5c\" (UID: \"cd2d26d1-c967-40d3-a5e4-9633bf2c4ed5\") " pod="odh-ai-gateway-infra/maas-api-85b64959d7-vwq5c" Oct 31 00:14:28.494059 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:28.493946 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd2d26d1-c967-40d3-a5e4-9633bf2c4ed5-maas-api-tls\") pod \"maas-api-85b64959d7-vwq5c\" (UID: \"cd2d26d1-c967-40d3-a5e4-9633bf2c4ed5\") " pod="odh-ai-gateway-infra/maas-api-85b64959d7-vwq5c" Oct 31 00:14:28.547966 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:28.547936 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrjw9\" (UniqueName: \"kubernetes.io/projected/cd2d26d1-c967-40d3-a5e4-9633bf2c4ed5-kube-api-access-jrjw9\") pod \"maas-api-85b64959d7-vwq5c\" (UID: \"cd2d26d1-c967-40d3-a5e4-9633bf2c4ed5\") " pod="odh-ai-gateway-infra/maas-api-85b64959d7-vwq5c" Oct 31 00:14:28.626434 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:28.626408 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85b64959d7-vwq5c" Oct 31 00:14:28.999869 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:28.999845 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-85b64959d7-vwq5c"] Oct 31 00:14:29.001003 ip-10-0-130-177 kubenswrapper[4952]: W1031 00:14:29.000973 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd2d26d1_c967_40d3_a5e4_9633bf2c4ed5.slice/crio-cd0ce60a87c45396d22c71260468280af534ab10985e6a21cbcfd8d13c03c89c WatchSource:0}: Error finding container cd0ce60a87c45396d22c71260468280af534ab10985e6a21cbcfd8d13c03c89c: Status 404 returned error can't find the container with id cd0ce60a87c45396d22c71260468280af534ab10985e6a21cbcfd8d13c03c89c Oct 31 00:14:29.137715 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:29.137690 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85b64959d7-vwq5c" event={"ID":"cd2d26d1-c967-40d3-a5e4-9633bf2c4ed5","Type":"ContainerStarted","Data":"cd0ce60a87c45396d22c71260468280af534ab10985e6a21cbcfd8d13c03c89c"} Oct 31 00:14:30.145630 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:30.145512 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85b64959d7-vwq5c" event={"ID":"cd2d26d1-c967-40d3-a5e4-9633bf2c4ed5","Type":"ContainerStarted","Data":"20fd712a0eec28ff016be206fc16375c8bf9cffb4452b85073f9932e2411588b"} Oct 31 00:14:30.146010 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:30.145738 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-85b64959d7-vwq5c" Oct 31 00:14:30.261372 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:30.261321 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-85b64959d7-vwq5c" podStartSLOduration=1.41394667 podStartE2EDuration="2.26130698s" totalImagesPullingTime="847.36031ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:30.260810677 +0000 UTC m=+4089.404640882" watchObservedRunningTime="2026-10-31 00:14:30.26130698 +0000 UTC m=+4089.405137239" Oct 31 00:14:30.482454 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:30.482379 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:30.800278 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:30.800204 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:33.729511 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:33.729471 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:33.954434 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:33.954395 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:35.245027 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:35.244998 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:35.461851 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:35.461813 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:36.155817 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:36.155785 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-85b64959d7-vwq5c" Oct 31 00:14:36.323437 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:36.323408 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-84b8bf64d7-djfcl"] Oct 31 00:14:36.323851 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:36.323615 4952 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-84b8bf64d7-djfcl" podUID="7ae58c73-211f-4a73-b0f7-8e1d5c092b1a" containerName="maas-api" containerID="cri-o://846ffd94c3debb6227bb5894c9c71aebd7af187fea0ca708efd969687c021c67" gracePeriod=30 Oct 31 00:14:36.571820 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:36.571791 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84b8bf64d7-djfcl" Oct 31 00:14:36.665610 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:36.665581 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ae58c73-211f-4a73-b0f7-8e1d5c092b1a-kube-api-access-vj6mj\" (UniqueName: \"kubernetes.io/projected/7ae58c73-211f-4a73-b0f7-8e1d5c092b1a-kube-api-access-vj6mj\") pod \"7ae58c73-211f-4a73-b0f7-8e1d5c092b1a\" (UID: \"7ae58c73-211f-4a73-b0f7-8e1d5c092b1a\") " Oct 31 00:14:36.665743 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:36.665655 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ae58c73-211f-4a73-b0f7-8e1d5c092b1a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ae58c73-211f-4a73-b0f7-8e1d5c092b1a-maas-api-tls\") pod \"7ae58c73-211f-4a73-b0f7-8e1d5c092b1a\" (UID: \"7ae58c73-211f-4a73-b0f7-8e1d5c092b1a\") " Oct 31 00:14:36.665743 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:36.665734 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ae58c73-211f-4a73-b0f7-8e1d5c092b1a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ae58c73-211f-4a73-b0f7-8e1d5c092b1a-openshift-service-ca\") pod \"7ae58c73-211f-4a73-b0f7-8e1d5c092b1a\" (UID: \"7ae58c73-211f-4a73-b0f7-8e1d5c092b1a\") " Oct 31 00:14:36.665878 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:36.665761 4952 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ae58c73-211f-4a73-b0f7-8e1d5c092b1a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ae58c73-211f-4a73-b0f7-8e1d5c092b1a-metrics-certs\") pod \"7ae58c73-211f-4a73-b0f7-8e1d5c092b1a\" (UID: \"7ae58c73-211f-4a73-b0f7-8e1d5c092b1a\") " Oct 31 00:14:36.666157 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:36.666125 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ae58c73-211f-4a73-b0f7-8e1d5c092b1a-openshift-service-ca" pod "7ae58c73-211f-4a73-b0f7-8e1d5c092b1a" (UID: "7ae58c73-211f-4a73-b0f7-8e1d5c092b1a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:14:36.668013 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:36.667984 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ae58c73-211f-4a73-b0f7-8e1d5c092b1a-maas-api-tls" pod "7ae58c73-211f-4a73-b0f7-8e1d5c092b1a" (UID: "7ae58c73-211f-4a73-b0f7-8e1d5c092b1a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:36.668164 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:36.668069 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ae58c73-211f-4a73-b0f7-8e1d5c092b1a-kube-api-access-vj6mj" pod "7ae58c73-211f-4a73-b0f7-8e1d5c092b1a" (UID: "7ae58c73-211f-4a73-b0f7-8e1d5c092b1a"). InnerVolumeSpecName "kube-api-access-vj6mj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:14:36.668659 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:36.668641 4952 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ae58c73-211f-4a73-b0f7-8e1d5c092b1a-metrics-certs" pod "7ae58c73-211f-4a73-b0f7-8e1d5c092b1a" (UID: "7ae58c73-211f-4a73-b0f7-8e1d5c092b1a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:36.766857 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:36.766783 4952 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ae58c73-211f-4a73-b0f7-8e1d5c092b1a-maas-api-tls\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:14:36.766857 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:36.766812 4952 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ae58c73-211f-4a73-b0f7-8e1d5c092b1a-openshift-service-ca\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:14:36.766857 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:36.766821 4952 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ae58c73-211f-4a73-b0f7-8e1d5c092b1a-metrics-certs\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:14:36.766857 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:36.766831 4952 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj6mj\" (UniqueName: \"kubernetes.io/projected/7ae58c73-211f-4a73-b0f7-8e1d5c092b1a-kube-api-access-vj6mj\") on node \"ip-10-0-130-177.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:14:37.186587 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:37.186546 4952 generic.go:356] "Generic (PLEG): container finished" podID="7ae58c73-211f-4a73-b0f7-8e1d5c092b1a" containerID="846ffd94c3debb6227bb5894c9c71aebd7af187fea0ca708efd969687c021c67" exitCode=0 Oct 31 00:14:37.186764 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:37.186620 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84b8bf64d7-djfcl" event={"ID":"7ae58c73-211f-4a73-b0f7-8e1d5c092b1a","Type":"ContainerDied","Data":"846ffd94c3debb6227bb5894c9c71aebd7af187fea0ca708efd969687c021c67"} Oct 31 00:14:37.186764 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:37.186622 4952 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84b8bf64d7-djfcl" Oct 31 00:14:37.186764 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:37.186664 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84b8bf64d7-djfcl" event={"ID":"7ae58c73-211f-4a73-b0f7-8e1d5c092b1a","Type":"ContainerDied","Data":"683abc5482517a00abd61f15765a1a221313cb722c39cd80a5cf4dfbc436f099"} Oct 31 00:14:37.186764 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:37.186688 4952 scope.go:122] "RemoveContainer" containerID="846ffd94c3debb6227bb5894c9c71aebd7af187fea0ca708efd969687c021c67" Oct 31 00:14:37.198227 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:37.198210 4952 scope.go:122] "RemoveContainer" containerID="846ffd94c3debb6227bb5894c9c71aebd7af187fea0ca708efd969687c021c67" Oct 31 00:14:37.198540 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:14:37.198518 4952 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"846ffd94c3debb6227bb5894c9c71aebd7af187fea0ca708efd969687c021c67\": container with ID starting with 846ffd94c3debb6227bb5894c9c71aebd7af187fea0ca708efd969687c021c67 not found: ID does not exist" containerID="846ffd94c3debb6227bb5894c9c71aebd7af187fea0ca708efd969687c021c67" Oct 31 00:14:37.198590 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:37.198550 4952 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"846ffd94c3debb6227bb5894c9c71aebd7af187fea0ca708efd969687c021c67"} err="failed to get container status \"846ffd94c3debb6227bb5894c9c71aebd7af187fea0ca708efd969687c021c67\": rpc error: code = NotFound desc = could not find container \"846ffd94c3debb6227bb5894c9c71aebd7af187fea0ca708efd969687c021c67\": container with ID starting with 846ffd94c3debb6227bb5894c9c71aebd7af187fea0ca708efd969687c021c67 not found: ID does not exist" Oct 31 00:14:37.350963 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:37.350931 4952 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-84b8bf64d7-djfcl"] Oct 31 00:14:37.353671 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:37.353652 4952 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-84b8bf64d7-djfcl"] Oct 31 00:14:38.970783 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:38.970750 4952 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ae58c73-211f-4a73-b0f7-8e1d5c092b1a" path="/var/lib/kubelet/pods/7ae58c73-211f-4a73-b0f7-8e1d5c092b1a/volumes" Oct 31 00:14:40.652746 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:40.652711 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:43.231825 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:43.231792 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:14:45.488280 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:45.488242 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:50.499186 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:50.499154 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:51.142577 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:14:51.142539 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:14:55.612247 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:14:55.612210 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:00.873137 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:15:00.873105 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:05.858903 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:15:05.858857 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:08.509375 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:15:08.509308 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:08.790221 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:15:08.790102 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:09.758952 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:15:09.758923 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:09.981506 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:15:09.981471 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:10.582588 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:15:10.582554 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:14.817749 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:15:14.817717 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:16.333502 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:15:16.333441 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:16.617630 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:15:16.617600 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:17.595132 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:15:17.595086 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:18.779045 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:15:18.779008 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:19.399395 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:15:19.399345 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:19.874421 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:15:19.874388 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:21.139465 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:15:21.139433 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:15:21.143416 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:15:21.143384 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:15:23.406744 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:15:23.406706 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:23.641599 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:15:23.641563 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:24.958436 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:15:24.958401 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:25.185307 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:15:25.185276 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:26.595658 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:15:26.595629 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:26.812087 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:15:26.812004 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:27.039107 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:15:27.038971 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:31.665814 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:15:31.665782 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:33.128170 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:15:33.128074 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:15:34.207696 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:15:34.207658 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:36.585506 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:15:36.585470 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:41.450579 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:15:41.450540 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:43.062614 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:15:43.062583 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:46.587550 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:15:46.587511 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:51.141819 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:15:51.141782 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:15:51.365079 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:15:51.365015 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:56.173234 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:15:56.173196 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:00.955426 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:16:00.955332 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:05.784460 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:16:05.784424 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:10.557687 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:16:10.557655 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:15.358066 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:16:15.358007 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:20.174051 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:16:20.174007 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:21.140541 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:16:21.140505 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:16:21.625101 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:16:21.625065 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft_91651ce2-4673-4ced-b26d-e94ff8829d11/extract/0.log" Oct 31 00:16:21.635279 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:16:21.635259 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/2.log" Oct 31 00:16:21.641352 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:16:21.641335 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 31 00:16:21.652735 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:16:21.652718 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft_91651ce2-4673-4ced-b26d-e94ff8829d11/extract/0.log" Oct 31 00:16:21.663238 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:16:21.663217 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/2.log" Oct 31 00:16:21.668554 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:16:21.668538 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 31 00:16:24.966146 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:16:24.966102 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:26.917361 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:16:26.917327 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:28.509636 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:16:28.509596 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:28.737210 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:16:28.737182 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:29.768634 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:16:29.768595 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:34.545281 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:16:34.545245 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:34.792857 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:16:34.792822 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:39.459804 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:16:39.459770 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:44.384703 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:16:44.384672 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:49.268683 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:16:49.268616 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:51.140735 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:16:51.140696 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:16:54.169794 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:16:54.169755 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:59.057794 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:16:59.057762 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:04.146711 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:17:04.146677 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:09.147604 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:17:09.147570 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:13.972454 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:17:13.972422 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:15.171087 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:17:15.171031 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:15.398001 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:17:15.397970 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:18.677016 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:17:18.676985 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:21.140927 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:17:21.140880 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:17:27.986365 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:17:27.986328 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:29.510197 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:17:29.510164 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:29.788138 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:17:29.788059 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:30.561573 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:17:30.561538 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:34.212693 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:17:34.212657 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:17:34.952832 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:17:34.952802 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:36.668432 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:17:36.668387 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:38.545543 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:17:38.545496 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:44.698941 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:17:44.698905 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:17:51.142268 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:17:51.142216 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:17:56.141999 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:17:56.141963 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:18:17.197188 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:18:17.197155 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:18:21.140563 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:18:21.140523 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:18:35.244428 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:18:35.244387 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:35.472183 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:18:35.472153 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:47.140575 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:18:47.140541 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-cf46b9688-9vzlq_ee7ad485-5d24-49cc-8072-0211b6ace77f/authorino/0.log" Oct 31 00:18:48.010212 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:18:48.010181 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hj2mm"] Oct 31 00:18:51.142731 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:18:51.142680 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:18:58.841930 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:18:58.841874 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-p47kn_656eacbe-1345-4cdd-b7c7-efa60412553d/manager/0.log" Oct 31 00:18:59.980929 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:18:59.980827 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-bxg8r_990ec266-ac95-4e83-a20e-503e8b5ee15e/server/0.log" Oct 31 00:19:00.519805 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:19:00.519751 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-69fdv_d5770149-5467-499e-8814-e52899bc3edc/proxy/0.log" Oct 31 00:19:01.527643 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:19:01.527610 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-85b64959d7-vwq5c_cd2d26d1-c967-40d3-a5e4-9633bf2c4ed5/maas-api/0.log" Oct 31 00:19:02.642766 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:19:02.642738 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-p4jb6_d6322b74-8f5a-46c5-98fc-0640e3a060a1/postgres/0.log" Oct 31 00:19:04.152169 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:19:04.152140 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-cf46b9688-9vzlq_ee7ad485-5d24-49cc-8072-0211b6ace77f/authorino/0.log" Oct 31 00:19:04.603027 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:19:04.602996 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-pkx96_8f8b6a0c-282f-45f4-a2ea-500a5cf81dd4/manager/0.log" Oct 31 00:19:04.819666 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:19:04.819641 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-pmgl5_5f95de50-95f0-4e61-a2a0-deacaece6891/manager/0.log" Oct 31 00:19:05.034714 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:19:05.034626 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-75gxr_fb7db2c7-13a2-44b6-83f3-b5966cff4343/kuadrant-console-plugin/0.log" Oct 31 00:19:05.570233 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:19:05.570211 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-hj2mm_52beb09a-acb7-4f24-a70b-195f1939ba32/limitador/0.log" Oct 31 00:19:05.806333 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:19:05.806299 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-9hxjd_e6751225-439f-4088-a474-183cd2d75baa/manager/0.log" Oct 31 00:19:06.598158 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:19:06.598132 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-56c7694f852x7dr_ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe/istio-proxy/0.log" Oct 31 00:19:08.551550 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:19:08.551500 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-6flbq_3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f/istio-proxy/0.log" Oct 31 00:19:09.302351 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:19:09.302311 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-dz26x_f5421fcf-649c-477a-b487-68c52d23f09b/istio-proxy/1.log" Oct 31 00:19:09.771131 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:19:09.771093 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-v4ktn_fd2831d3-e0d4-4a4b-a209-d5cb09632279/istio-proxy/0.log" Oct 31 00:19:09.995575 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:19:09.995534 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct_a82aebc2-0acd-4643-9481-7f1c0c6ddf35/istio-proxy/0.log" Oct 31 00:19:10.658305 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:19:10.658278 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5dd6f58df-tctqk_5f6c1ad7-10c4-4995-ae18-315d38184360/router/0.log" Oct 31 00:19:11.150342 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:19:11.150303 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-dz26x_f5421fcf-649c-477a-b487-68c52d23f09b/istio-proxy/0.log" Oct 31 00:19:14.223052 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:19:14.223014 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-cf46b9688-9vzlq_ee7ad485-5d24-49cc-8072-0211b6ace77f/authorino/0.log" Oct 31 00:19:21.140259 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:19:21.140223 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:19:24.357663 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:19:24.357629 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-p47kn_656eacbe-1345-4cdd-b7c7-efa60412553d/manager/0.log" Oct 31 00:19:25.040009 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:19:25.039986 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-bxg8r_990ec266-ac95-4e83-a20e-503e8b5ee15e/server/0.log" Oct 31 00:19:25.570630 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:19:25.570588 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-69fdv_d5770149-5467-499e-8814-e52899bc3edc/proxy/0.log" Oct 31 00:19:26.375818 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:19:26.375790 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-85b64959d7-vwq5c_cd2d26d1-c967-40d3-a5e4-9633bf2c4ed5/maas-api/0.log" Oct 31 00:19:27.287868 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:19:27.287836 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-p4jb6_d6322b74-8f5a-46c5-98fc-0640e3a060a1/postgres/0.log" Oct 31 00:19:28.406612 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:19:28.406567 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-cf46b9688-9vzlq_ee7ad485-5d24-49cc-8072-0211b6ace77f/authorino/0.log" Oct 31 00:19:28.863736 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:19:28.863691 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-pkx96_8f8b6a0c-282f-45f4-a2ea-500a5cf81dd4/manager/0.log" Oct 31 00:19:29.086881 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:19:29.086854 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-pmgl5_5f95de50-95f0-4e61-a2a0-deacaece6891/manager/0.log" Oct 31 00:19:29.302904 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:19:29.302839 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-75gxr_fb7db2c7-13a2-44b6-83f3-b5966cff4343/kuadrant-console-plugin/0.log" Oct 31 00:19:29.775754 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:19:29.775733 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-hj2mm_52beb09a-acb7-4f24-a70b-195f1939ba32/limitador/0.log" Oct 31 00:19:30.000847 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:19:30.000822 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-9hxjd_e6751225-439f-4088-a474-183cd2d75baa/manager/0.log" Oct 31 00:19:30.579352 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:19:30.579321 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-56c7694f852x7dr_ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe/istio-proxy/0.log" Oct 31 00:19:32.453301 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:19:32.453267 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-6flbq_3a6cdb7b-cb0d-44fd-a06e-607b947ddb3f/istio-proxy/0.log" Oct 31 00:19:33.200556 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:19:33.200526 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-dz26x_f5421fcf-649c-477a-b487-68c52d23f09b/istio-proxy/1.log" Oct 31 00:19:33.680549 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:19:33.680514 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-v4ktn_fd2831d3-e0d4-4a4b-a209-d5cb09632279/istio-proxy/0.log" Oct 31 00:19:33.900692 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:19:33.900669 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-599fb7fb95-bkbct_a82aebc2-0acd-4643-9481-7f1c0c6ddf35/istio-proxy/0.log" Oct 31 00:19:34.573558 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:19:34.573528 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5dd6f58df-tctqk_5f6c1ad7-10c4-4995-ae18-315d38184360/router/0.log" Oct 31 00:19:35.574521 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:19:35.574493 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:51.141077 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:19:51.141010 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:20:21.142698 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:20:21.142659 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:20:22.330264 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:20:22.330229 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:35.918150 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:20:35.918118 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:49.440182 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:20:49.440145 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:49.668204 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:20:49.668172 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:51.142115 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:20:51.142080 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:20:59.037293 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:20:59.037264 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7vrpr_322e3a9b-911f-4cd8-8f43-71d56152e917/global-pull-secret-syncer/0.log" Oct 31 00:21:00.219250 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:00.219224 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-lthcb_8a7f82b8-115d-43ed-a7a2-cb869a7e48e0/connection-checker/0.log" Oct 31 00:21:00.639198 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:00.639170 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-mpbb4_624d7fe2-4960-48ba-b711-3d85e4bf0687/connection-checker/0.log" Oct 31 00:21:01.060763 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:01.060690 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-szm9n_67e30e6a-a537-4e2c-a484-f15281828fec/connection-checker/0.log" Oct 31 00:21:01.508795 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:01.508769 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" Oct 31 00:21:02.860212 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:02.860173 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-177.us-west-1.compute.internal_f343176fefb75130c5aad85365de861f/haproxy/0.log" Oct 31 00:21:07.996978 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:07.996942 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-cf46b9688-9vzlq_ee7ad485-5d24-49cc-8072-0211b6ace77f/authorino/0.log" Oct 31 00:21:08.309920 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:08.309838 4952 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-d5xbs/perf-node-gather-daemonset-vl8cp"] Oct 31 00:21:08.310261 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:08.310249 4952 cpu_manager.go:414] "containerMap: removing container" podUID="7ae58c73-211f-4a73-b0f7-8e1d5c092b1a" containerName="maas-api" Oct 31 00:21:08.310310 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:08.310263 4952 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ae58c73-211f-4a73-b0f7-8e1d5c092b1a" containerName="maas-api" Oct 31 00:21:08.310356 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:08.310347 4952 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ae58c73-211f-4a73-b0f7-8e1d5c092b1a" containerName="maas-api" Oct 31 00:21:08.313449 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:08.313432 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-vl8cp" Oct 31 00:21:08.322097 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:08.322076 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-pkx96_8f8b6a0c-282f-45f4-a2ea-500a5cf81dd4/manager/0.log" Oct 31 00:21:08.395422 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:08.395395 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b415d20-8af0-44e5-a1ef-1b0b998189fb-proc\") pod \"perf-node-gather-daemonset-vl8cp\" (UID: \"4b415d20-8af0-44e5-a1ef-1b0b998189fb\") " pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-vl8cp" Oct 31 00:21:08.395422 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:08.395425 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b415d20-8af0-44e5-a1ef-1b0b998189fb-podres\") pod \"perf-node-gather-daemonset-vl8cp\" (UID: \"4b415d20-8af0-44e5-a1ef-1b0b998189fb\") " pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-vl8cp" Oct 31 00:21:08.395624 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:08.395543 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k5kz\" (UniqueName: \"kubernetes.io/projected/4b415d20-8af0-44e5-a1ef-1b0b998189fb-kube-api-access-2k5kz\") pod \"perf-node-gather-daemonset-vl8cp\" (UID: \"4b415d20-8af0-44e5-a1ef-1b0b998189fb\") " pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-vl8cp" Oct 31 00:21:08.395624 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:08.395568 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b415d20-8af0-44e5-a1ef-1b0b998189fb-lib-modules\") pod \"perf-node-gather-daemonset-vl8cp\" (UID: \"4b415d20-8af0-44e5-a1ef-1b0b998189fb\") " pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-vl8cp" Oct 31 00:21:08.395624 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:08.395590 4952 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b415d20-8af0-44e5-a1ef-1b0b998189fb-sys\") pod \"perf-node-gather-daemonset-vl8cp\" (UID: \"4b415d20-8af0-44e5-a1ef-1b0b998189fb\") " pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-vl8cp" Oct 31 00:21:08.414887 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:08.414856 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d5xbs\"/\"kube-root-ca.crt\"" Oct 31 00:21:08.414887 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:08.414862 4952 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d5xbs\"/\"openshift-service-ca.crt\"" Oct 31 00:21:08.415112 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:08.415096 4952 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-d5xbs\"/\"default-dockercfg-xrnc4\"" Oct 31 00:21:08.417401 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:08.417380 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d5xbs/perf-node-gather-daemonset-vl8cp"] Oct 31 00:21:08.496167 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:08.496139 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k5kz\" (UniqueName: \"kubernetes.io/projected/4b415d20-8af0-44e5-a1ef-1b0b998189fb-kube-api-access-2k5kz\") pod \"perf-node-gather-daemonset-vl8cp\" (UID: \"4b415d20-8af0-44e5-a1ef-1b0b998189fb\") " pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-vl8cp" Oct 31 00:21:08.496355 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:08.496177 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b415d20-8af0-44e5-a1ef-1b0b998189fb-lib-modules\") pod \"perf-node-gather-daemonset-vl8cp\" (UID: \"4b415d20-8af0-44e5-a1ef-1b0b998189fb\") " pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-vl8cp" Oct 31 00:21:08.496355 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:08.496198 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b415d20-8af0-44e5-a1ef-1b0b998189fb-sys\") pod \"perf-node-gather-daemonset-vl8cp\" (UID: \"4b415d20-8af0-44e5-a1ef-1b0b998189fb\") " pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-vl8cp" Oct 31 00:21:08.496355 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:08.496236 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b415d20-8af0-44e5-a1ef-1b0b998189fb-proc\") pod \"perf-node-gather-daemonset-vl8cp\" (UID: \"4b415d20-8af0-44e5-a1ef-1b0b998189fb\") " pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-vl8cp" Oct 31 00:21:08.496355 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:08.496260 4952 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b415d20-8af0-44e5-a1ef-1b0b998189fb-podres\") pod \"perf-node-gather-daemonset-vl8cp\" (UID: \"4b415d20-8af0-44e5-a1ef-1b0b998189fb\") " pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-vl8cp" Oct 31 00:21:08.496355 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:08.496298 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b415d20-8af0-44e5-a1ef-1b0b998189fb-lib-modules\") pod \"perf-node-gather-daemonset-vl8cp\" (UID: \"4b415d20-8af0-44e5-a1ef-1b0b998189fb\") " pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-vl8cp" Oct 31 00:21:08.496355 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:08.496335 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b415d20-8af0-44e5-a1ef-1b0b998189fb-sys\") pod \"perf-node-gather-daemonset-vl8cp\" (UID: \"4b415d20-8af0-44e5-a1ef-1b0b998189fb\") " pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-vl8cp" Oct 31 00:21:08.496355 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:08.496347 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b415d20-8af0-44e5-a1ef-1b0b998189fb-proc\") pod \"perf-node-gather-daemonset-vl8cp\" (UID: \"4b415d20-8af0-44e5-a1ef-1b0b998189fb\") " pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-vl8cp" Oct 31 00:21:08.496613 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:08.496385 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b415d20-8af0-44e5-a1ef-1b0b998189fb-podres\") pod \"perf-node-gather-daemonset-vl8cp\" (UID: \"4b415d20-8af0-44e5-a1ef-1b0b998189fb\") " pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-vl8cp" Oct 31 00:21:08.552610 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:08.552577 4952 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k5kz\" (UniqueName: \"kubernetes.io/projected/4b415d20-8af0-44e5-a1ef-1b0b998189fb-kube-api-access-2k5kz\") pod \"perf-node-gather-daemonset-vl8cp\" (UID: \"4b415d20-8af0-44e5-a1ef-1b0b998189fb\") " pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-vl8cp" Oct 31 00:21:08.627203 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:08.627174 4952 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-vl8cp" Oct 31 00:21:08.747642 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:08.747613 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-pmgl5_5f95de50-95f0-4e61-a2a0-deacaece6891/manager/0.log" Oct 31 00:21:08.998683 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:08.998661 4952 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d5xbs/perf-node-gather-daemonset-vl8cp"] Oct 31 00:21:09.000164 ip-10-0-130-177 kubenswrapper[4952]: W1031 00:21:09.000141 4952 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4b415d20_8af0_44e5_a1ef_1b0b998189fb.slice/crio-29ca7b3b4c1e26f8bc5a4c2677208a4245bc446a3631d2526b2f6fd8b63b3ad1 WatchSource:0}: Error finding container 29ca7b3b4c1e26f8bc5a4c2677208a4245bc446a3631d2526b2f6fd8b63b3ad1: Status 404 returned error can't find the container with id 29ca7b3b4c1e26f8bc5a4c2677208a4245bc446a3631d2526b2f6fd8b63b3ad1 Oct 31 00:21:09.166899 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:09.166871 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-75gxr_fb7db2c7-13a2-44b6-83f3-b5966cff4343/kuadrant-console-plugin/0.log" Oct 31 00:21:09.496352 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:09.496319 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-vl8cp" event={"ID":"4b415d20-8af0-44e5-a1ef-1b0b998189fb","Type":"ContainerStarted","Data":"eb15fd5781aaeaf0fd1383b402d406aa76c5b102ab8d17447eecdba685399819"} Oct 31 00:21:09.496577 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:09.496378 4952 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-vl8cp" event={"ID":"4b415d20-8af0-44e5-a1ef-1b0b998189fb","Type":"ContainerStarted","Data":"29ca7b3b4c1e26f8bc5a4c2677208a4245bc446a3631d2526b2f6fd8b63b3ad1"} Oct 31 00:21:09.496577 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:09.496418 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-vl8cp" Oct 31 00:21:09.610829 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:09.610774 4952 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-vl8cp" podStartSLOduration=1.610761617 podStartE2EDuration="1.610761617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:09.609849399 +0000 UTC m=+4488.753679605" watchObservedRunningTime="2026-10-31 00:21:09.610761617 +0000 UTC m=+4488.754591821" Oct 31 00:21:10.015331 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:10.015297 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-hj2mm_52beb09a-acb7-4f24-a70b-195f1939ba32/limitador/0.log" Oct 31 00:21:10.472583 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:10.472541 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-9hxjd_e6751225-439f-4088-a474-183cd2d75baa/manager/0.log" Oct 31 00:21:15.513758 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:15.513731 4952 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-vl8cp" Oct 31 00:21:16.060868 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:16.060832 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-dh6nh_813fd5cb-7fda-45f4-b9d0-6d8040e7f7e5/cluster-monitoring-operator/0.log" Oct 31 00:21:21.142605 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:21.142573 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l2rht_17ed56a5-3a2a-494b-aaa0-a9032aa5f265/node-exporter/0.log" Oct 31 00:21:21.143198 ip-10-0-130-177 kubenswrapper[4952]: E1031 00:21:21.143171 4952 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lfbzq" containerName="extractor" Oct 31 00:21:21.558783 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:21.558699 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l2rht_17ed56a5-3a2a-494b-aaa0-a9032aa5f265/kube-rbac-proxy/0.log" Oct 31 00:21:21.665184 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:21.665160 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft_91651ce2-4673-4ced-b26d-e94ff8829d11/extract/0.log" Oct 31 00:21:21.675871 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:21.675848 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/2.log" Oct 31 00:21:21.682059 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:21.682007 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 31 00:21:21.696296 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:21.696270 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hrkft_91651ce2-4673-4ced-b26d-e94ff8829d11/extract/0.log" Oct 31 00:21:21.707101 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:21.707077 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-kztwm_e9a89b14-a8e8-45f2-84bd-dba21617499c/console-operator/2.log" Oct 31 00:21:21.712862 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:21.712840 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ts59n_20eb8eea-5fd8-42ab-bd78-8abc77f84eb2/ovn-acl-logging/0.log" Oct 31 00:21:21.978962 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:21.978925 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l2rht_17ed56a5-3a2a-494b-aaa0-a9032aa5f265/init-textfile/0.log" Oct 31 00:21:27.208821 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:27.208783 4952 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9sp4c_e4d82728-b04b-4914-93d2-fce161acae36/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:31.654566 ip-10-0-130-177 kubenswrapper[4952]: I1031 00:21:31.654539 4952 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-56c7694f852x7dr_ad91d15d-ceb9-464a-8a8a-aa9bf61eb7fe/istio-proxy/0.log"